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ABSTRACT 

This document consists of a workshop presentation on 
curriculum development, design, specification, and assessment in the 
engineering classroom. Ten sessions focus on: (1) the format and 
purpose of the workshop, which is designed to help instructors 
develop an understanding of the basic principles of curriculum and 
instruction and the ability to apply these principles to the 
development, design, specification, and assessment of a curriculum or 
course; (2) cognitive aspects of learning science; (3) the state of 
instruction in the engineering sciences and definitions of learning; 
(4) the elements of learning systems, learning styles, and taxonomies 
of learning; (5) educational states evaluation exercises; (6) 
educational goals and competencies; (7) past, present, and future 
classroom structures; (8) the classification of outcomes and 
competencies, as well as curriculum and course design; (9) the 
transformation of goals and outcomes into competencies; and (10) 
sequen'^ing competencies. An appendix contains outlines of various 
cognitive models. A supplementary volume contains sample competency 
matrices, a guide to self -evaluation and documentation of educational 
states, a guide to the documentation of technical work, learning 
structures for students, excerpts from "Cognitive Aspects of Learning 
Science*' (Jose Mestre) , a collection of information on change in 
education, and four papers on intellectual development and student 
assessment. (MDM) 
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Curriculum Development, Design, Specification and Assessment 



Session 0. (10 minutes / 1 0 minutes total ) 

Getting Started 

Forming Groups 

Focus on Facilitator Signal 

Issue Bin 

Reflection (the Academic Journal) 
Code of Cooperation 



Session 1 . (30 minutes / 40 minutes total ) 

A Private Universe ( video tape ) 

Question: Determine at least TWO issues illustrated in the tape 

which are relevant to your Engineering classroom? 
Active Leaming Delivery: Formulate-Share-Listen-Create (Think-Pair-Share) 



Session 2. 



( 5 minutes / 45 minutes total ) 



Why are we (me and my absent team member) here? 
How did we get here? 
Cognition Defined 
Cognitive Science Domains 



Session 3. 



( 10 minutes / 65 minutes total ) 



Leaming Systems Elements 
Leaming Styles 

Blooms Taxonomy ( Cognitive and Affective ) 
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Session 4. (100 minutes / 165 minutes total ) 

Educational States Evaluation 
Active Learning Delivery: Jigsaw 



BREAK ( 60 to 90 minutes but the next day is better ) 



Session 5. ( 60 minutes / 60 minutes total ) 

Educational Goals, Curriculum Outcomes and Competencies 
Developing an Assessment Instrument 



Session 6. (10 minutes / 70 minutes total ) 

Classroom Structures ( Yesterday, Today and Tomorrow ) 



Session 7. ( 20 minutes / 90 minutes total ) 

Classifying Outcomes, Competencies 
Curriculum (Course) Development Model 

Competencies versus Level of Learning and Degree of Internalization 
The Competency Matrix 
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Session 8. ( 25 minutes / 1 1 5 minutes total ) 

Transforming Goals Into Competencies; Building the Tree ( Part I ) 



Session 9. ( 40 minutes / 155 minutes total ) 

Sequencing the Tree 

Competencies, 
Levels of Leaming, 
Degrees of Internalization 
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r Learning Styles 



9 



Abilities by Style 



Sensing / T>lf!king, 35% 





Gooj^t working with 
and^membering details 

/ 

ib\e to speak and write 
directly to the point 

/ Approaches task In an 
organized and sequential manner 



Intuiting / Thinking, 20% 



idling 



Spontaneous and op^ to 
impulse, does what feels^ood 

Able to express 
personal feelings 

Aware of others' feelings and 

makes Judgements based \ 
on personal likes and dislikes 



\\ 



Takes trme to 
plan and contemplate 
consequences of actions 




\Able to organize and 
s^thesize Information 

Weights tlk^eyidence and risks 
judgememb€(sed on logic 



Intuiting /Feeling, 10% 

Gc jd at Intepreting 
facts and details tosee 
the broader picture 

Able to express Ideas 
in new and unusual v^s 

/ 

Approachesj^ks 
in a variety^f ways 
or In an^jBJS^ratory manner 



Hanson, Silver, Strong and Associates, 

The Thoughtful Education People', 

34 Washington Road , Princeton Jet. , NJ 08550 

(609) 799-6300 Phone (609) 799-6301 FAX 
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Bloorn's Taxonomy : 
Comitive Domain (1950) 

1. Knowledge of : 

1.1 Specifics 

1.2 Ways & Means of Dealing with Specifics 

2. Comprehension 

2.1 Translation 

2.2 Interpretation 

2.3 Extrapolation 

3. Application (Out of Context) 

4. Analysis of : 

4.1 Elements 

4.2 Relationships 

4.3 Organizational Principles 

5. Synthesis 

5.1 Production of a Unique Communication 

5.2 Production of a Plan, 

or Proposed set of Operations 

5.3 Derivation of a set of Abstract Relations 

6. Evaluation ... Judgment in Terms of : 

6.1 Internal Evidence 

6.2 External Criteria 



blooms 



. ^ Slide 23 

Blobrn's Taxonomy : 
Affejkive Domain (1964) 

1. Receiving (Attending) 

1.1 Awareness 

1.2 Willingness to Receive 

1.3 Controlled or Selected Attention 

2. Responding 

2.1 Acquiescence in Responding 

2.2 Willingness to Respond 

2.3 Satisfaction in Response 

3. Valuing 

3.1 Acceptance of a Value 

3.2 Preference for a Value 

3.3 Commitment 

4. Organization 

4.1 Conceptualization of a Value 

4.2 Organization of a Value System 

5. Characterization by a Value 

or Value Complex 

5.1 Generalized Set 

5.2 Characterization 
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1.0 
RECEIVING 


2 WILUNGNESS TO RECEIVE 








CO CONTROLLED OR SELECTED 
^ ATTENTION 














2.0 

RESPONDING 


ACQUIESCENCE IN 
«^ RESPONDING 
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^ WILUNGNESS TO RESPOND 
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^ SATISFACTION IN RESPONSE 
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3.0 
VALUING 


acceptance of a value 
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H 
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^ PREFERENCE FOR A VALUE 




K 


^ i 








^ COMMITMENT 


JUSTMENT 






Q 






4.0 

ORGANIZATION 


CONCEPTUALIZATION OF A 
VALUE 






r ^ 




C4 ORGANIZATION OF A VALUE 
^ SYSTEM 










5.0 

CHARACTERIZATION 
BY A VALUE 
COMPLEX 


generalized set 






^ characterization 







The range of meaning typical of commonly uted affective temfis measured against the Taxonomy continuum. 
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ifying Outcomes 
- Taxonomies 

I ❖ Bloom et al (1948 - 1964) for EACH TOPIC 

• 6 Levels of Learning 

- knowledge, comprehension, 
I application, analysis, synthesis, 

I evaluation 

• 5 Degrees of Internalizaion (intrinsic 
I motivation in Quality) 

I - receiving, responding, valuing, 

I organization, characterization 

❖ Canelos & Catchen (1989) 

• fractal learning, concept learning, rule 
|. learning, problem solving 

I <^ Dreyfus & Dreyfus (1983) 

I • novice, advanced beginner, competent 
performer, proficient performer, expert 

' ❖ Merrill's (1983) for constructivist learning 
I environments 

I ❖Gagne's (1987) for constructivist learning 
environments 
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A Condensed Version of the Cognitive Domain 
of the Taxonomy of Educational Objectives (3) 



1.00 Knowledge 

Knowledge, as defined here, involves the recall of specifics and universals, the recall of 
methods and processes, or the recall of a pattern, structure, or setting. For measurement 
purposes, the recall situation involves little more than bringing to mind the appropriate 
materials. Although some alteration of the material may be required, this is a relatively 
minor part of the task.. The knowledge objectives emphasize most the psychological 
processes of remembering. The process of relating is also involved in that a knowledge 
test situation requires the organization and reorganization of a problem such that it will 
furnish the appropriate signals and cues for the information and knowledge the individual 
processes. To use an analogy, if one thinks of the mind as a file, the problem in a 
knowledge test situation is that of finding in the problem or task the appropriate signals, 
cues, and clues which will most effectively bring out whatever knowledge is filed or stored. 



1.10 Knowledge Of Specifics 

The recall of specific and isolable bits of information. The emphasis is one symbols with 
concrete referents, This material, which is at a very low level of abstraction, may be 
thought of as the elements, from which more complex and abstract forms of knowledge 
are built. 



1.11 Knowledge of Terminoiogy 

Knowledge of the referents for specific symbols (verbal and nonverbal). This may include 
knowledge of the most generally accepted symbol referent, knowledge of the variety of 
symbols which may be used for single referent, or knowledge of the referent most 
appropriate to a given use of a symbol. 

To define technical terms by giving their attributes, properties, or relations. 
Familiarity with a large number of words in their common range of meanings. 



1.12 Knowledge of Specific Facts 

Knowledge of dates, events, persons, places, etc. This may include very precise and 
specific information such as the specific day or exact magnitude of a phenomenon. It may 
also include approximate or relative information such as an approximate time period or the 
general order of magnitude of a phenomenon. 

The recall of major facts about particular cultures. 

The possession of a minimum knowledge about the organisms studied in the 
laboratory. 



1.20 Knowledge Of Ways And Means Of Dealing With Specifics 

Knowledge of the ways of organizing, studying, judging, and criticizing. This includes the 
methods of inquiry, the chronological sequences, and the standards of judgment within a 
field as well as the patterns of organization through which the areas of the fields 
themselves are determined and internally organized. This knowledge is at an 
intermediate level of abstraction between specific knowledge on the one hand and 
knowledge of universals on the other. It does not so much demand the activity of the 
student in using the materials as it does a more passive awareness of their nature. 



1.21 Knowledge of Conventions 

Knowledge of characteristics ways of treating and presenting ideas and phenomena. For 
purposes of communication and consistency, workers in a field employ usages, styles, 
practices, and forms which best suit their purposes and/or which appear to suit best the 
phenomena with which they deal. It should be recognized that although these forms and 
conventions are likely to be set up on arbitrary, accidental, or authoritative bases, they are 
retained because of the general agreement or concurrence of individuals concerned with 
the subject, phenomena, or problem. 

Familiarity with the forms and conventions of the major types of works; e.g., verse, 
plays, scientific papers, etc. 

To make pupils conscious of correct form and usage in speech and writing. 
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1.22 Knowledge of Trends and Sequences 

Knowledge of the processes, directions, and movements of pi cnomena witli respect to 
time. 

Understanding of the continuity and development of American culture as exemplified in 
Arn©ric3n lit© 

Knowledge of the basic trends underlying the development of public assistance 
programs. 



1.23 Knowledge of Classifications and Categories 

Knowledge of the classes, sets, divisions, and arrangements which are regarded as 
fundamental for a given subject field, purpose, argument, or problem. 

To recognize the area encompassed by various kinds of problems or materials. 
Becoming familiar with a range of types of literature. 



1.24 Knowledge of Criteria 

Knowledge of the criteria by which facts, principles, opinions, and conduct are tested or 
judged. 

Familiarity with criteria for judgment appropriate to the type of work and the purpose for 
which it is read. 

Knowledge of criteria for the evaluation of recreational activities, 



1.25 Knowledge of Methodology 

Knowledge of the methods of inquiry, techniques, and procedures employed in a particular 
subject field as well as those employed in investigating particular problems and 
phenomena. The emphasis here is on the individual's knowledge of the method rather 
than his ability to use the method. 

Knowledge of scientific methods for evaluating health concepts. 

The student shall know the methods of attack relevant to the kinds of problems of 

concern to the social sciences. 
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1.30 Knowledge Of The Universals And Abstractions In A Field 

Knowledge of the major schemes and pattems by which phenomena and ideas are 
organized. These are the large structures, theories, and generalizations v;hich dominate a 
subject field or which are quite generally used in studying phenomena or solving 
problems. These are at the highest levels of abstractions and complexity. 



1.31. Knowledge of Principles and Generalizations 

Knowledge of particular abstractions which summarize observations of phenomena. 
These are the abstractions which are of value in explaining, describing, predicting, or in 
determining the most appropriate and relevant action or direction to be taken. 

Knowledge if the important principles by which our experience with biological 
phenomena is summarized. 

The recall of major generalizations about particular cultures. 



1.32 Knowledge of Theories and Structures 

Knowledge of the body of principles and generalizations together with their interrelations 
which present a clear, rounded, and systematic view of a complex phenomenon, problem, 
or field. These are the most abstract formulation, and they can be used to show the 
interrelation and organization of a great range of specifics. 

The recall of major theories about particular cultures. 

Knowledge of a relatively complete formulation of the theory of evolution. 
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Intellectual Abilities And Skills 
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Abilities and skills refer to organized modes of operation and generalized techniques for 
dealing with materials and problems. The materials and problems may be of such a 
nature that little or no specialized and technical information is required. Such information 
as is required can be assumed to be part of the individual's general fund of knowledge. 
Other problems may require specialized and technical information at a rather high level 
such that specific knowledge and skill in dealing with the problem and the materials are 
required. The abilities and skills objectives emphasize the mental processes of organizing 
and reorganizing material to achieve a particular purpose. The materials may be given or 
remembered. 



2.00 Comprehension 

This represents the lowest level of understanding. It refers to a type of understanding or 
apprehension such that the individual knows what Is being communicated and can make 
use of the material or idea being communicated without necessarily relating it to other 
material or seeing its fullest implications. 



2.10 Translation 

Comprehension as evidenced by the care and accuracy with which the communication is 
paraphrased or rendered from one language or form of communication to another. 
Translation is judged on the basis of faithfulness and accuracy; that is, on the extent to 
which the material in the original communication is preserved although the form of the 
communication has been altered. 

The ability to understand non literal statements (metaphor, symbolism, irony, 
exaggeration). 

Skill in translating mathematical verbal material into symbolic statements and vice 
versa. 
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2.20 Interpretation 

The explanation or summarization of a communication. Whereas translation involves an 
objective part-for-part rendering of a communication, interpretation involves a reordering, 
rearrangement, or new view of the material. 

The ability to grasp the thought of the work as a whole at any desired level of 
generality. 

The ability to interpret various types of social data. 



2.30 Extrapolation 

The extension of trends or tendencies beyond the given data to determine implications, 
consequences, corollaries, effects, etc., which are in accordance with the conditions 
described in the original communication. 

The ability to deal with the conclusion of a work in terms of the inference made from 

the explicit statements. 

Skill in predicting continuation of trends. 



3.00 Application 

The use of abstractions in particular and concrete situation. The abstractions may be in 
the form of general ideas, rules of procedures, or generalized methods. The abstractions 
may also be technical principles, ideas, and theories which must be remembered and 
applied. 

Application to the phenomena discussed in one paper of the scientific terms or 
concepts used in other papers. 

The ability to predict the probable effect of a change in a factor on a biological situation 
previously at equilibrium. 
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4.00 Analysis 



The breakdown of a communication into its constituent elements or parts such that the 
relative hierarchy of ides is made clear and/or the relations between the ideas expressed 
are made explicit. Such analyses are intended to clarify the communication, to indicate 
how the communication is organized, and the way in which it manages to convey its 
effects, as well as its basis and arrangement. 



4. 10 Analysis Of Elements 

Identification of the elements included in a communication. 

The ability to recognize unstated assumptions. 
Skill in distinguishing facts from hypotheses. 



4.20 Analysis Of Relationships 

The connections and interactions between elements and parts of a communication. 

Ability to check the consistency of hypotheses with given information and assumption. 
Skill in comprehending the interrelationships among the ideas in a passage. 



4.30 Analysis Of Organizational Principles 



The organization, systematic arrangement, and structure which hold the communication 
together. This includes the "explicit" as well as "implicit" structure. It includes the bases, 
necessary arrangement, and mechanics which make the communication a unit. 

The ability to recognize form and pattern in literary or artistic works as a means of 
understanding their meaning. 

Ability to recognize the general techniques used in persuasive materials, such as 
advertising, propaganda, etc. 
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5.00 Synthesis 

The putting together of elements and parts so as to form a whole. This Involves the 
process of working with pieces, parts, elements, etc., and arranging and combining them 
in such a way as to constitute a pattem or structure not clearly there before. 



5.70 Production Of A Unique Communication 

The development of a communication in which the writer or speaker attempts to convey 
ideas, feelings, and/or experiences to others. 

Skill in writing, using an excellent organization of ideas and statements. 
Ability to tell a personal experience effectively. 



5.20 Production Of A Plan, Or Proposed Set Of Operations 

The development of a plan of work or the proposal of a plan of operations. The plan 
should satisfy requirements of the task which may be given to the student or which he 
may develop for himself. 

Ability to propose ways to testing hypotheses. 

Ability to plan a unit of instruction for a particular teaching situation. 



5.30 Derivation Of A Set Of Abstract Relations 

The development of a set of abstract relations either to classify or explain particular data 
or phenomena, or the deduction of propositions and relations from a set of basic 
propositions or symbolic representations. 

Ability to formulate appropriate hypotheses based upon an analysis of factors involved, 
and to modify such hypotheses in the light of new factors and considerations. 
Ability to make mathematical discoveries and generalizations. 
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6.00 Evaluation 

Judgments about the value of material and methods for given purposes. Quantitative and 
qualitative judgments about the extent to which material and methods satisfy criteria. Use 
of a standard of appraisal. The criteria may be those determined by the student or those 
which are given to him. 



6.10 Judgments In Terms Of Internal Evidence 

Evaluation of the accuracy of a communication from such evidence as logical accuracy, 
consistency, and other internal criteria. 

Judging by intemal standards, the ability to assess general probability of accuracy in 
reporting facts from the care given to exactness of statement, documentation, proof, 
etc. 

The ability to indicate logical fallacies in arguments. 



6.20 Judgments In Terms Of External Criteria 

Evaluation of material with reference to selected or remembered criteria. 

The comparison of major theories, generalizations, and facts about particular cultures. 
Judging by extemal standards, the ability to compare a work with the highest known 
standards in its field-especially with other works of recognized excellence. 
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(1) Each sub category is followed by illustrative educational objectives soiected from the literature. 
(3) Bloom. David R.. et al.. Taxonomy of Educational Objectives, Longman, 1964. p. 176 - 185. 
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A Condensed Version of the Affective Domain 
of the Taxonomy of Educational Objectives (1) 



1 .0 Receiving (Attending) 

At this level we are concerned that the learner be sensitized to the existence of certain 
phenomena and stimuli; that is, that he be willing to receive or to attend to them. This is 
clearly the first and crucial step if the leamer is to be properly oriented to leam what the 
teacher intends that he will. To indicate that this is the bottom rung of the ladder, 
however, is not at all to imply that the teacher is starting de novo. Because of previous 
experience (formal or informal), the student brings to each situation a point of view or set 
which may facilitate or hinder his recognition of the phenomena to which the teacher is 
trying to sensitize him. 

The category of Receiving has been divided into three subcategories to indicate three 
different levels of attending to phenomena. While the division points between the 
subcategories are arbitrary, the subcategories do represent a continuum. From an 
extremely passive position or role on the part of the leamer, where the sole responsibility 
for the evocation of the behavior rests with the teacher-that is, the responsibility rests with 
him for "capturing" the student's attention-the continuum extends to a point at which the 
leamer directs his attention, at least at a semiconscious level, toward the preferred stimuli. 



1.1 Awareness 

Awareness is almost a cognitive behavior. But unlike Knowledge, the lowest level of the 
cognitive domain, we are not so much concemed with a memory of, or ability to recall, an 
item of fact as we are that, given appropriate opportunity, the leamer will merely be 
conscious of something-that he take into account a situation, phenomena, object, or stage 
of affairs. Like Knowledge it does not imply an assessment of the qualities or nature of 
the stimulus, but unlike Knowledge it does not necessarily imply attention. There can be 
simple awareness without specific discrimination or recognition of the objective 
characteristics of the object, even though these characteristics must be deemed to have 
an effect. The individual may not be able to verbalize the aspects of the stimulus which 
cause the awareness. 

Develops awareness of aesthetic factors in dress, furnishings, architecture, city design, 
good art, and the like. 

Develops some consciousness of color, form, arrangement, and design in the objects 
and structures around him and in descriptive or symbolic representations of people, 
things, and situation. (2) 



ERIC 



155 



I 
I 



1.2 Willingness to Receive 

In this category we have come a step up the ladder but are still dealing with what appears 
to be cognitive behavior. At a minimunn level, we are here describing the behavior of 
being willing to tolerate a given stimulus, not to avoid it. Like Awareness, it involves a 
neutrality or suspended judgment toward the stimulus. At this level of the continuum the 
teacher is not concemed that the student seek it out, nor even, perhaps, that in an 
environment crowded with many other stimuli the learner will necessarily attend in a field 
with relatively few competing stimuli, the learner is not actively seeking to avoid it. At best, 
he is willing to take notice of the phenomenon and give it his attention. 

Attends (carefully) when others speak-in direct conversation, on the telephone, in 
audiences. 

Appreciation (tolerance) of cultural patterns exhibited by individuals from other groups- 
religious, social, political, economic, national, etc. 
Increase in sensitivity to human need and pressing social problems. 



1.3 Controlled or Selected Attention 

At a somewhat higher level we are concemed with a new phenomenon, the differentiation 
of a given stimulus into figure and ground at a conscious or perhaps semiconscious level- 
the differentiation of aspects of a stimulus which is perceived as clearly marked off from 
adjacent impressions. The perception is still without tension or assessment, and the 
student may not know the technical terms or symbols with which to describe it correctly or 
precisely to others. In some instances it may refer not so much to the selectivity of 
attention as to the control of attention, so that when certain stimuli are present they will be 
attended to. There is an element of the leamer's controlling the attention here, so that the 
favored stimulus is selected and attended to despite competing and distracting stimuli. 

Listens to music with some discrimination as to its mood and meaning and with some 
recognition of the contributions of various musical elements and instruments to the 
total effect. 

Alertness toward human values and judgments on life as they are recorded in 
literature. 
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2.0 Responding 



At this level we are concerned with responses which go beyond merely attending to the 
phenomenon. The student is sufficiently motivated that he Is not just 1 .2 Willing to attend, 
but perhaps it is correct to say that he is actively attending. As a first stage in a "learning 
by doing" process the student is committing himself in some small measure to the 
phenomena involved. This is a very low level of commitment, and we would not say at this 
level that this was "a value of his" or that he had "such and such an attitude." These terms 
belong to the next higher level that we describe. But we could say that he is doing 
something with or about the phenomenon besides merely perceiving it, as would be true 
at the next level below this of 1 .2 Controlled or selected attention. 

This is the category that many teachers will find best describes their "interest" objectives. 
Most commonly we use the term to indicate the desire that a child become sufficiently 
involved in or committed to a subject, phenomenon, or activity that he will seek it out and 
gain satisfaction from working with it or engaging in it. 



2.1 Acquiescence In Responding 

We might use the word "obedience" or "compliance" to describe this behavior. As both of 
these terms indicate, there is a passiveness so far as the initiation of the behavior is 
concemed, and the stimulus calling for this behavior is not subtle. Compliance is perhaps 
a better term than obedience, since there is more of the element of reaction to a 
suggestion and less of the implication of resistance or yielding unwillingly. The student 
makes the response, but he has not fully accepted the necessity for doing so. 

Willingness to comply with health regulations. 
Obeys the playground regulation. 



2.2 Willingness To Respond 

The key to this level is in the term "willingness," with its implication of capacity for 
voluntary activity. There is the implication that the learner is sufficiently committed^ to 
exhibiting the behavior that he does so not just because of a fear or punishment, but "on 
his own" or voluntarily. It may help to note that the element of resistance or of yielding 
unwillingly, which is possibly present at the previous level, is here replaced with consent or 
proceeding from one's own choice. 

Acquaints himself with significant current issues in intemational, political, social, and 
economic affairs through voluntary reading and discussion. 

Acceptance of responsibility for his won health and for the protection of the health of 
others. 
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2.3 Satisfaction In Response 

The additional element in the step beyond the Willingness to respond level, the consent, 
the assent to responding, or the voluntary response, is that the behavior is accompanied 
by a feeling of satisfaction, an emotional response, generally of pleasure, zest, or 
enjoyment. The location of this category in the hierarchy has given is a great deal of 
difficulty. Just where in the process of intemalization the attachment of an emotional 
response, kick, or thrill to a behavior occurs has been hard to determine. For that matter 
there is some uncertainty as to whether the level of intemalization at which it occurs may 
not depend on the particular behavior. We have even questioned whether it should be a 
category. If our stmcture is to be a hierarchy, then each category should include the 
behavior in the next level below it. The emotional component appears gradually through 
the range of internalization categories. The attempt to specify a given position in the 
hierarchy as the one at which the emotional component is added is doomed to failure. 

The category is arbitrarily placed at this point in the hierarchy where it seems to appear 
most frequently and where it is cited as or appears to be an important component of the 
objectives at this level on the continuum. The category's inclusion at this point serves the 
pragmatic purpose of reminding us of the presence of the emotional component and its 
value in the building of affective behaviors. But it should not be thought of as appearing 
and occurring at this one point in the continuum and thus destroying the hierarchy which 
we are attempting to build. 

Enjoyment of self-expression in music and in arts and crafts as another means of 

personal enrichment. 

Finds pleasure in reading for recreation. 

Takes pleasure in conversing with many different kinds of people. 



3.0 Valuing 

This is the only category headed by a term which is in common use in the expression of 
objectives by teachers. Further, it is employed in its usual sense: that a thing, 
phenomenon, or behavior has worth. This abstract concept of worth is in part a result of 
the individuars own valuing or assessment, but it is much more a social product that has 
been slowly internalized or accepted and has come to be used by the student as his own 
criterion of worth. 

Behavior categorized at this level is sufficiently consistent and stable to have taken on the 
characteristics of a belief or an attitude. The learner displays this behavior with sufficient 
consistency in appropriate situations that he comes to be perceived as holding a value. At 
this level, we are not concemed with the relationships among values but rather with the 
intemalization of a set of specified, ideal, values. Viewed from another standpoint, the 
objectives classified here are the prime stuff from which the conscience of the individual is 
developed into active control of behavior. 

This category will be found appropriate for many objectives that use the term "attitude" (as 
well as, of course, "value"). 

An important element of behavior characterized by Valuing is that it is motivated, not by 
the desire to comply or obey, but by the individual's commitment to the underlying value 
guiding the behavior. 



3.1 Acceptance Of A Value 

At this level we are concemed with the ascribing of worth to a phenomenon, behavior, 
object, etc. The term "belief, which is defined as "the emotional acceptance of a 
proposition or doctrine upon what one implicitly considers adequate ground" (English and 
English, 1958, p. 64), describes quite well what may be thought of as the dominant 
characteristic here. Beliefs have varying degrees of certitude. At this lowest level of 
Valuing we are concemed with the lowest levels of certainty; that is, there is more of a 
readiness to re-evaluate one's position than at the higher levels. It is a position that is 
somewhat tentative. 

One of the distinguishing characteristics of this behavior is consistency of response to the 
class of objects, phenomena, etc. with which the belief or attitude is identified. It is 
consistent enough so that the person is perceived by others as holding the belief or value. 
At the level we are describing here, he is both sufficiently consistent that others can 
identify the value, and sufficiently committed that he is willing to be so identified. 

Continuing desire to develop the ability to speak and write effectively. 
Grows in his sense of kinship with human beings of all nations. 
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3.2 Preference For A Value 

The provision for this subdivision arose out of a feeling that there were objectives that 
expressed a level of internalization between the mere acceptance of a value and 
connmitment or conviction in the usual connotation of deep involvement in an area. 
Behavior at this level implies not just the acceptance of a value to the point of being willing 
to be identified with it, but the individual is sufficiently committed to the value to pursue it, 
to seek it out, to want it. 

Assumes responsibility for drawing reticent members of a group into conversation. 
Deliberately examines a variety of viewpoints on controversial issues with a view to 
forming opinions about them. 

Actively participates in arranging for the showing of contemporary artistic efforts. 



3.3 Commitment 

Belief at this level involves a high degree of certainty. The ideas of "conviction" and 
"certainty beyond a shadow of a doubt" help to convey further the level of behavior 
intended. In some instances this may border on faith, in the sense of it being a firm 
emotional acceptance of a belief upon admittedly non rational grounds. Loyalty to a 
position, group, or cause would also be classified here. 

The person who displays behavior at this level is clearly perceived as holding the value. 
He acts to further the thing valued in some way, to extend the possibility of his developing 
it, to deepen his involvement with it and with the things representing it. He tries to 
convince others and seeks convarts to his cause. There is a tension here which needs to 
be satisfied; action is the result of an aroused need or drive. There is a real motivation to 
act out the behavior. 

Devotion to those ideas and ideals which are the foundations of democracy. 
Faith in the power of reason and in methods of experiment and discussion. 



4.0 Organization 

As the learner successively internalizes values, he encounters situation for which more 
than one value is relevant. Thus necessity arises for (a) the organization of the values 
into a system, (b) the determination of the interrelationships among them, and (c) the 
establishment of the dominant and pervasive ones. Such a system is built gradually, 
subject to change as new values are incorporated. This category is intended as the 
proper classification for objectives which describe the beginnings of the building of a value 
system. It is subdivided into two levels, since a prerequisite to interrelating is the 
conceptualization of the value in a form which permits organization. Conceptualization 
forms the first subdivision in the organization process. Organization of a value system the 
second. 

While the order of the two subcategories seems appropriate enough with reference to one 
another, it is not so certain that 4.1 Conceptualization of a value is properly placed as the 
next level above 3.3 Commitment. Conceptualization undoubtedly begins at an earlier 
level for some objectives. Like 2.3 satisfaction in response, it is doubtful that single 
completely satisfactory location for this category can be found. Positioning it before 4.2 
Organization of a value system appropriately indicates a prerequisite of such a system. It 
also calls attention to a component of affective growth that occurs at least by this point on 
the continuum but may begin earlier. 



4.1 Conceptualization Of A Value 

In the previous category, 3.0 Valuing, we noted that consistency and stability are integral 
characteristics of the particular value or belief. At this level (4.1) the quality of abstraction 
or conceptualization is added. This permits the individual to see how the value relates to 
those that he already holds or to new ones that he is coming to hold. 

Conceptualization will be abstract, and in this sense it will be symbolic. But the symbols 
need not be verbal symbols. Whether conceptualization first appears at this point on the 
affective continuum is a moot point, as noted above. 

Attempts to identify the characteristics of an art object which he admires. 

Forms judgments as to the responsibility of society for conserving human and material 

resources. 
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4.2 Organization Of A Value System 

Objectives properly classified here are those which require the learner to bring together a 
complex of values, possibly disparate values, and to bring these into an ordered 
relationship with one another. Ideally, the ordered relationship will be one which is 
harmonious and internally consistent. This is, of course, the goal of such objectives, 
which seek to have the student formulate a philosophy of life. In actuality, the integration 
may be something less than entirely harmonious. More likely the relationgtfip is better 
described as a kind of dynamic equilibrium which is, in part, dependent upon those 
portions of the environment which are salient at any point in time. In many instances the 
organization of values may result in their synthesis into a new value or value complex of a 
higher order. 

Weighs altemative social policies and practices against the standards of the public 
welfare rather than the advantage of specialized and narrow interest groups. 
Develops a plan for regulating his rest in accordance with the demands of his activities. 



5.0 Characterization By A Value Or Value Complex 

At this level of intemalization the values already have a place in the individual's value 
hierarchy, are organized into some kind of intemally consistent system, have controlled 
the behavior of the individual for sufficient time that he has adapted to behaving this way; 
and an evocation of the behavior no longer arouses emotion or affect except when the 
individual is threatened or challenged. 

The individual acts consistently in accordance with the values he has intemalized at this 
level, and our concem is to indicate two things; (a) the generalization of this control to so 
much of the individual's behavior that he is described and cha'-acterized as a person by 
these pervasive controlling tendencies, and (b) the integration of these beliefs, ideas, and 
attitudes into a total philosophy or world view. These two aspects constitute the 
subcategories. 
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5.1 Generalized Set 

The generalized set is that which gives an internal consistency to the system of attitudes 
and values at any particular moment. It is selective responding at a very high level. It is 
sometimes spoken of as a determining tendency, an orientation toward phenomena, or a 
predisposition to act in a certain way. The generalized set is a response to highly 
generalized phenomena. It is a persistent and consistent response to a family of related 
situation or objects. It may often be an unconscious set which guides action without 
conscious forethought. The generalized set may be thought of as closely related to the 
idea of an attitude cluster, where the commonality is based on behavioral characteristics 
rather than the subject or object of the attitude. A generalized set is a basic orientation 
which enables the individual to reduce and order the complex world about him and to act 
consistently and effectively in it. 

Readiness to revise judgments and to change behavior in the light of evidence. 
Judges problems and issues in terms of situation, issues, purposes, and 
consequences involved rather than in terms fixed, dogmatic precepts or emotionally 
wishful thinking. 



5.2 Characterization 

This, the peak of the internalization process, includes those objectives which are broadest 
with respect both to the phenomena covered and to the range of behavior which they 
comprise. Thus, here are found those objectives which concern one's view of the 
universe, one's philosophy of life, one's Weltanschauung-a value system having as its 
object the whole of what is known or knowable. 

Objectives categorized here are more than generalized sets in the sense that they involve 
a greater inclusiveness and, within the group of attitudes, behaviors, beliefs, or ideas, an 
emphasis on intemal consistence. Though this intemal consistency may not always be 
exhibited behaviorally by the students toward whom the objective is directed, since we are 
categorizing teachers' objectives, this consistency feature will always be a component of 
Characterization objectives. 

As the title of the category implies, these objectives are so encompassing that they tend to 
characterize the individual almost completely. 

Develops for regulation of one's personal and civic life a code of behavior based on 
ethical principles consistent with democratic ideals 
Develops a consistent philosophy of life. 



(1) Bloom, David R., et al., Taxonomy of Educational Objectives, Longman, 1964, p. 176 - 185. 

(2) Illustrative objectives selected from tlie literature follow the description of each sub category. 
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Self Evaluation and Documentation of Educational States 



Evaluation 

In today's work environment it is essential that you strive to improve processes, products 
and yourself. However, before you can make suggestions for improvements, you must 
know the current status of the processes, products and yourself. Evaluation allows you to 
know where to make improvements, where the weak areas are which need to be improved 
When you graduate you will find that the self evaluation process presented in this guide will 
be as useful as the engineering science knowledge you have gained in your course work. 
Your education is the end product of an educational process and it is important to know the 
status of this education (product). Traditionally the evaluation of the status of your 
education has been done by course instructors who assign a grade to indicate the state ot 
your education. While it may be relatively easy to let instructors do this evaluation of your 
work (i e you do the assigned work, submit the work and wait to see what the instructor 
reports) in the long term you will need to be the person doing the evaluation. You cannot 
continually rely on someone else to tell you how you are doing; you must learn how to 
evaluate your own educational state so you can make the required improvements. You 
must take responsibility for your education. 

Before describing a method that can be used in self evaluation and documentation of 
educational states, it is necessary to more fully define what is meant by educational states. 

Educational States 

How can educational states be defined or characterized? Whiie there are many ways to do 
this one possible way is to characterize your educational state by the activities and actions 
of you and your teacher. Reflect back over your time in school; you should be able to 
recognize the gross changes which have taken place in your activities. Early on you 
learned facts and worked simple , single concept problems; towards the end you worked 
problems which combined many different concepts and skills. 

In the early 1950's a group of educational psychologists considered the problem of defining 
educational and developed a taxonomy of educational objectives^ . To quote from the 
Foreword of this work: 

It (the taxonomy) is intended to provide for classification of the goals of our 
educational system. It is expected to be of general help to all teachers, 
administrators, professionals specialists, and research workers who deal with 
curricular and evaluation problems. It is especially intended to help them discuss 
these problems with greater precision. 
These psychologists divided the problem up into three behavioral domains: the cognitive, 
dealing with the recall or recognition of knowledge and the development of intellectual 
abilities and skills; the affective, dealing with changes in interest, attitudes, and values, 
and the development of appreciation and adequate adjustment, and the psychomotor, 
dealing with the manipulative or motor-skill area.. 

The taxonomy (a handbook) published in the mid 50's dealt only with the cognitive domain 
(a second handbook dealing with the affective domain was published in 1964). The 



Taxonomy of Educational Objectives, Bloom et al, 1956, Longmans, Green and Co. 
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taxonomy for the cognitive domain had six major categories. In order of increasing 
complexity they were: Knowledge, Comprehension, Application, Analysis, Synthesis, and 
Evaluation. Each of these categories was characterized by a different set of abilities 
(behaviors) exhibited by a person operating in the category. 

In the late 80's and early 90's David Langford, in attempting to implement an important 
aspect of the quality culture (empowemient) into the classroom, recognized that this 
taxonomy could be used by the students as well as the teachers to determine where they 
were relative to these various objectives. Thus Langford proposed having the students use 
these educational objectives to do self evaluation. Langford renamed the objectives Levels 
of Learning, changed the name of Comprehension to Know - How, and developed 
summaries of the types of activities a student and teacher would do when they, the 
students, were operating at these various Levels of Learning^ . 

A modification to Langford's material on typical activities can be found in the first six pages 
of Appendix A. This material consists of answers to a set of six standard questions. The 
answers to the questions change as the Level of Leaming changes. For example, the 
answer to the question of How do I know I have reached this level? is answered as, / recall 
information, at the Knowledge Level but is answered as, / have the ability to put together 
parts and elements into a unified organization or whole which requires originah creative 
thinking, at the Synthesis Level. Read these answers over carefully, paying especially 
close attention to the first two and last two questions for each of the levels. 

As the psychologists pointed out in the 50's there is a bit more to defining an educational 
state than is covered by the Levels of Learning. The Levels of Learning concern only the 
cognitive domain of the educational state; they do not concern the affective domain, a 
domain which must also be assessed. Affective behavior is indicated by your willingness 
to take responsibility for your education, to put in the needed effort, to be interested in what 
is being leamed, to help others. Myron Tribus in several essays^ states that Character 
(which is in some ways a part of affective behavior) is one of the major categories of things 
which should be cultivated in a school or university. 

The last page of the material in Appendix A addresses the affective or character issue. 
This material is no where near as complete as the Level of Learning material, containing 
only some general affective traits and a few self awareness questions. 

The Self Evaluation Process 

Two different process are used to evaluate your educational state: one for the cognitive 
domain and one for the affective domain. 
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2 Total Quality Leaming Handbook, Langford Quality Education, 1992 

3 Quality Management in Education, and Total Quality Management in Schools of Business and of 
Engineering, Myron Tribus, Exergy. Inc., Hayward. CA 
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Cognitive Domain 

For the cognitive domain the self evaluation process is reasonably simple: 

1 . compare your current abilities or activities to a set of Abilities or Activities 
Exemplars'^ , exemplars defined by a hierarchy of learning, 

2. pick out the Exemplar which best fit your current abilities or activities, 

3. the selected Exemplar defines your current educational state. 

For this class you are to use the activities defined for the various Levels of Learning in 
Appendix A as the Exemplars. For example, if you are in Statics working single concept 
problems located at the end of a Section, you would be at the Know-How Level of Leaming 
for the current statics topic because what you are doing matches the type of activity a 
person at that Level of Leaming would be doing. On the other hand, if you were writing 
reports on the design of a bridge you would probably be at the Synthesis Level of Learning 
for Statics because creation of evocative reports is an activity which is done by a person at 
that Level of Learning. 

Affective Domain 

It is much easier to be self aware of your cognitive abilities or states than it is to be aware 
of your affective state. It is difficult to be an expert in your own behavior. To help bridge 
this difficulty the following process is proposed: 

1 . Select a periodic timing sequence (e.g., each day, each week, each problem); 

2. select a set of affective traits; 

3. at the end of each timing sequence and for each selected affective trait ask 
your self whether you exhibited the characteristic during the previous time 
period. 

4. If the answer for a particular trait is consistently yes then you are exhibiting 
that affective trait. 

For example suppose you want to document your interest (an affective trait) in Statics. 
You decide to evaluate this trait once a week on Friday afternoon by asking yourself "how 
many times during this last week have I discussed Statics with someone not in Statics?". If 
the answer is consistently three or more then you could feel confident that you were 
showing interest in Statics. 

Documentation of Educational States 

While it may be possible for you to fairly and correctly evaluate your actual educational 
state with only internal (i.e., in your brain) evidence, that will not be acceptable for this 
class. In this class you must supply documentation that you have actually reached the 
educational states you are claiming. The documentation processes you are to use are 
modification of the processes suggested by Langford. 



^ Exemplar - that which serves as a pattern, especially an ideal pattern 
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Documentation Vehicles 

The documentation process uses several different vehicles for recording and storing your 
work. You must understand what the each of these vehicles is used for and how you can 
generate and use them before you can effectively document the required documentation. 

A Portfolio or Design Notebook 

A portfolio is an organized collection of the personal technical work done in a class. It 
contains, in some logical sequence, all homework assignments, quizzes, tests, reports, 
projects, i.e., everything technical done during the semester. A student's portfolio 
becomes a collection of worked examples, examples the student can refer to in later 
classes (or when out of school) when he needs to review a topic done in the earlier class. 

Physically, portfolios are three hole loose leaf binders although an accordion file can also 
be used. Non loose leaf binders are not too practical because it is not easy to remove or 
insert material. There are any number of ways to organize the work in the portfolio but two 
minimum requirements are that: 

1 . it must be possible to refer to any specific page or section (i.e., all pages must be 
uniquely numbered), and 

2. if a reader opens the portfolio to a random location, it must be obvious which way she 
would go to get to a specified location (i.e., W follows M or 12 comes before 34). 

These two requirements can generally be met by numbering all pages with some type of 
hierarchical page numbering scheme (e.g., 11.2 is page two of section II which comes 
before section III and after section I). 

A design notebook is a special type of portfolio, one containing all the technical work 
related to a design project. Since design projects are often done by teams the design 
notebook will include work from a variety of people. 

All the technical work in the portfolio/notebook is expected to adhere to accepted standards 
for documentation of technical work (e.g., see Documentation of Technical Work - The 
Process and the Product by McNeill). 

Reflection Log 

The work contained in your portfolio documents all the activities you have undertaken 
during the semester. Some of these activities reflect Know-How, some reflect Evaluation. 
As previously mentioned, traditionally it was the course instructor's task to decide what 
Level of Leaming the work reflected; however, in the self analysis mode, this task of 
matching work to Levels of Learning is to be done by you. You must reflect on the 
technical work you have done, analyzing it to determine what Level of Learning is 
documented (demonstrated) by the material. 

Once youVe completed this reflection you must write a paragraph (or more) explaining the 
reasons why some set of your technical work shows you are functioning at a claimed Level 
of Learning. This phase of the documentation is done using a Reflection Log (see 
Appendix B for a sample log). 

The log need not look like the one shown in Appendix B but it must contain the same 
infonnation. The log must have an identifying entry number; it must be clear which 
competency category (see the next section on the Competency Matrix) and Level of 
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Learning are being addressed; it nnust be clear where the technical work being discussed 
is located; and finally the log must contain the paragraphs of reflection. You will generally 
find that the reflection for the higher Levels of Learning requires more space than is given 
on the form, in which case you can append the necessary additional pages to the log entry. 

Physically the log must follow the Competency Matrix. 

Competency Matrix 

In any class the instructor has a set of knowledge and skills (e.g., Engineering Design 
Process, Second Order Differential Equations, First Law of Thermodynamics, Teaming, 
etc.) which she wants to have the class learn. This set of knowledge and skills are know 
as the learning outcomes for the class. Learning outcomes are rather abstract and must 
be characterized by (i.e., defined in terms of) a number of more specific topics called 
competency categories. Depending on how specific the competency categories are, it may 
be possible (desirable) to further divide these competency categories. 

However it is not enough to just define the leaming outcomes and competency categories, 
the instructor must also decide what Level of Learning the students should reach for each 
of these items. This crazy quilt of things to be leamed and the levels they are to be 
learned to can be organized and presented in a Competency Matrix. 

The general design of the matrix is quite simple. Along the left side of the matrix are the 
general course Leaming Outcomes along with each Outcomes' more specific Competency 
Categories (and Sub-categories if they exist). Along the top of the matrix are the various 
Levels of Learning. Each cell in the matrix represents the intersection of a particular 
competency category or learning outcome and a particular Level of Leaming. The 
Competency Matrix for this class is in Appendix B. 

In looking at your Competency Matrix you will see black areas, gray areas, and white 
areas. The black areas reflect the Levels of Learning you are assumed to have reached 
when you start the class. The gray areas are the Levels of Learning that you are expected 
to achieve during the course of the semester. The white areas represent Levels of 
Learning which you may achieve but which are not ones explicitly desired to be achieved in 
the class. 

As a vehicle for documenting your educational state you will find that the matrix serves two 
purposes. 

1 . It shows the Levels of Learning you have achieved in each of the course's 
competency categories, and 

2. it shows where there is documentation that you have in fact mastered the competency 
category at the Level of Leaming shown. 

At the start of a semester, except for the black areas, your matrix is blank. During the 
semester you will make entries in each of the gray (and perhaps white) boxes in the matrix. 
These entries are pointers to the reflective discussion of the technical work which you feel 
supports your claim of being able to operate at some Level of Learning. What is actually 
put in these boxes will be explained in the next section. 
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The Competency Matrix belongs in tlie front of tlie portfolio followed by the Reflection 
Logs, Work Logs, and Run Charts. If you are using a design notebook all of this evaluation 
material (i.e., the matrix, logs, and run charts) belongs in a separate loose leaf binder 
called the Evaluation Notebook. 

Work Logs 

Work logs as the name implies, keep track of the work you have done. A sample, empty 
log, is shown in Appendix B. The log contains factual information related to your work. 
The log tel!s you when the work was done, how much time you spent on the work, where 
the work is located, and finally a very brief description of the work. 

Work logs are generally found just behind the Reflection Logs in your Evaluation Notebook. 
Run Charts 

It is often necessary to keep track of a quantity whose value changes with time; it is also 
often necessary to keep track of the time averaged value of this quantity . The time 
averaged value is known as the running average for the quantity. While this data can be 
logged in tables, trends are hard to see in tables and, thus, this data is generally also 
shown on a graph. Such graphs are called Run Charts. 

A sample Run Chart for class attendance is shown in Appendix B. The chart shows time 
along the horizontal axis (i.e., class number) and the value of attendance along the vertical 
axes (yes = 1 , no = 0). The data for each class is shown by the height of the bar while the 
running average is shown by the line. In the example shown the person missed class on 
the 3rd, 5th, and 1 1th day (bar has zero height). For the third class the running average 
dropped from 1 (perfect attendance) to 0.67 (attendance for two of three classes). The 
running average drops for each class not attended and slowly rises when classes are 
attended. You can see that the running average after twenty classes is about .86. 

The Documentation Process - Cognitive Domain 

The basic documentation process is a three step process. However, once you get the feel 
for this basic process you will most likely want to consider adding a fourth step (step 0) 
which is discussed at the end of this section. 

the basic process 

The following three steps should be completed for each class assignment. 
step 1 

Read the assigned problem and decide what work must be done. Work the problem. Be 
sure to follow accepted standards for the documentation of technical work, (i.e., explain the 
process you followed or used, show the results of following the process, and discuss the 
results of the process). Once the work is done and documented, place the work in an 
appropriate location in the portfolio or design notebook. 
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step 2 

Reflect on the work just completed: 

a) decide what course competency categories are involved in the work (look at 
the matrix to see the list of competency categories), and 

b) decide what Levels of Learning have been demonstrated (documented) in the 
work. 

Once this reflection is complete, go to your Reflection Log and make a new entry or entries 
for the work you just entered into your portfolio. You must make separate log entries for 
each competency category / Level of Learning pair (i.e., for each box in the Competency 
Matrix). The one exception is when a piece of work covers a number of competency 
categories, ail at the same Level of Learning in which case you may use a single log entry 
for all the categories. You may never use a single log entry when different Levels of 
Learning are being claimed, even when there is only one competency category. 

step 3 

Every competency category / Level of Learning pair in the Reflection Log corresponds to a 
box in the Competency Matrix. Starting with the first new log entry enter its Log Entry 
Number in the appropriate Competency Matrix box. Repeat this process until all of the new 
Reflection Log entries you just made in step 2 have been processed (i.e., logged in on the 
matrix). 

a sample 

For example, suppose you have been assigned to work problem 2.2 in Statics, which 
involves determining an unknown force acting on a simple beam. You work the problem 
(including all required documentation) and put it in your Design Notebook at pages 2.5-2,6. 
Upon reflection you decide that the work shows that you have Know-How in the 
competency categories of free body diagrams and equilibrium. You go to your Reflection 
Log and find that the next entry will be number 12 and so you fill out the log (Log Entry 
Number: 12; Competency Category(s): free body diagram, equilibrium] Level of Leaming: 
Know How, Location in Design Notebook: pages 2.5 & 2.6\ Reflection: Problem 2.2 
requested a free body diagram and told me to use the idea of equilibrium of forces to 
determine the unknown force. Since the problem pretty much told me what to do and I was 
able to do it this is evidence of Know-i-iow but not Application L of L The documentation 
process is finished by going to your Competency Matrix and putting 12 (the log entry 
number) in the box for (free body diagram, Know-How) and (equilibrium,Know-How). 

an enhanced documentation process 

As mentioned at the beginning of this section there is probably a fourth, initial, step which 
you will want to add to this process once you begin to see how the basic process works. 

As currently proposed the documentation process has you work the problem before 
reflecting on what Levels of Learning have been demonstrated. Would it not make sense 
to think about the Levels of Learning before actually doing the work? If you did this 
wouldn't you produce technical work which more clearly showed evidence of the Level of 
Learning claimed? If the answers to these questions are yes then you will want to add 
Step 0 to your process. 
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step 0 

Read the problem and look at your Competency Matrix. Based on the problem and on 
what competency categories have not yet been addressed in the matrix (i.e., which gray 
boxes are still empty) decide what competency category / Level of teaming pairs could be 
addressed by the problem. Before going to Step 1 consider how you can best show the 
Levels of Learning you are hoping to demonstrate with the completion of the assignment. 

The Documentation Process - Affective Domain 

You can document your affective traits as follows: 
step 1 

For each assignment or class related outside of class activity make an entry in your Work 
Log. 

step 2 

At the start of each class notes down (yes or no) whether you were on time, were at class, 
had done the assigned reading, and had done the assigned homework. 

step 3 

Each Sunday evening, using the data collected at the start of each class and the data in 
your Work Log update the following Run Charts: 

1 . Class Attendance 

2. On Time to Class 

3. C»ass Preparation (Reading) 

4. Class Assignments 

5. Average Hours per Week Outside of Class Spent on Class 
When Can You Claim Mastery? 

How many problems do you have to work to show you are at the Analysis Level of 
Learning; how many for Know-How? What value for the running averages on the Run 
Charts shows you consistently demonstrate the affective trait? When can you claim 
mastery? 

The question of mastery has not been addressed up to this point but it is an important 
element in this evaluation process. Mastery means that not only have you reached an 
educational state for some competency category but that you will be able to operate at this 
level indefinitely (i.e., you will not slip back to some lower level). Mastery at the Application 
Level of Learning for Differential Calculus means that you routinely recognize when 
Differential Calculus is required in solving problems. 

Mastery does not mean that you have instant recall or the ability to instantly solve the 
problem at the Level of Learning claimed. No one has everything they have mastered 
available instantaneously. Mastery does mean that you can, relatively easily, retrieve the 
necessary information (e.g., from your Portfolio) to solve the problem at the specified Level 
of Learning. 
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Self Evaluation and Documentation of Educational States 

Further, mastery does not mean that you must be aware of what you are doing. Mr. House 
of Hewlett Packard Co. has suggested that as you gain mastery of a skill or subject area 
you pass through four stages of awareness: unconscious incompetent (you don't know 
that you don't know), conscious incompetent (you know that you don't know), conscious 
competent (you know that you know), and unconscious competent (you don't know that 
you know). You would like to get to the fourth state (i.e., be able to solve problems without 
thinking about how you are solving problems) but the third state is perfectly acceptable. 

There is no pat or definitive answer to the questions posed in the first paragraph. In fact 
the answer generally changes from subject to subject. For this class the table below 
shows hovi/ many problems must be worked to demonstrate mastery. 



Mastery Minimums 



Educational State 


Mastery Minimum Requirements 


Knowledge 


100 % correct on a quiz of at least three questions 


Know-How 


one example 


Application 


three examples 


Analysis 


two examples which between them coverall the competency 
areas covered by the Learning Outcome 


Synthesis 


same as Analysis 


Evaluation 


same as Analysis 


Run Charts (except for 
hours per week) 


the running average should be at or above 0.9 


Run Chart (hours per 
week) 


the running average should be at or above nine hours 



If these minimums are followed then your matrix will show mastery of a category when the 
matrix box has one log entry for Knowledge, or Know-How; two log entries for Analysis, 
Synthesis, and Evaluation; three log entries for Application and your Run Charts show your 
running averages to be at or above 0.9 (9 hours for work per week). 
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Reflection Log 
XXX YYY 



Log Entry Number: 



Competency Category(s): 



Level of Learning Claimed: 



Location in Portfolio or Design Notebook of Supporting Work: 
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Location in Portfolio or Design Notebook of Supporting Work: 
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An Alternative Way to Fill Out Matrix 

A number of you seem to be having difficulty in adding the discussion which explains 
why a certain piece of work documents some level of learning - it does not seem to be 
a natural piece of the work. With this in mind let me suggest an alternative method of 
filling out the matrix. 

step 1 

Read the problem and decide what competency areas could be addressed by the 
problem and determine what level of learning is possible. Work the problem in a 
manner which shows you have mastered the competency area(s) at the level decided 
upon. Be sure to completely document the work, i.e., explain the process you followed, 
show the results of following the process, and discuss the results of the process (all 
three parts are required). Once the work is done and documented place it at some 
appropriate location in your portfolio. 

step 2 

Go to your list of work, a set of blank pages located just behind your matrix, and add an 
entry to this list of work, using the next consecutive number. The entry must include the 
following: 

a) name of assignment just completed 

b) page number in portfolio where assignment can be found 

c) competency area(s) addressed along with level of learning claimed 

d) a paragraph explaining why the work is proof of achieving the level of learning 
claimed 

step 3 

On you matrix, at the appropriate location(s) (competency area and level of learning) 
put the list number you used in step 2. 

For example, if you did some work which you felt proved Know-How for j, you would put 
the work in your portfolio (say at page 45). Next you would update your list of work by 
adding a new entry. If the last entry were item 32 then this new entry would be item 33. 
You would enter in the required information and then go to the matrix and enter in 33 at 
all the places you have claimed by the work. 

This indirect indexing scheme has the advantage of removing the discussion of why the 
work proves a certain level of leaming from the work and concentrating all this material 
just behind the matrix. 

If you remove an item from the list of work just remove it and its number; there is no 
need to change any of the following numbers. 
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DocumentationThe material presented here has been developed over 
the last seventeen years at Arizona State University in the cou 

The goal of any documentation effort is to present the work that has been done in a 
form that makes it clearly understandable to other people. As an engineer, you spend 
many hours growing the design. Concepts are developed and compared, models are 
created and run, information is sought out from various resources, etc. All of this work 
needs to be recorded in a way that will permit you, or any technically competent person, 
to recreate the process that was followed or understand why the design is as it is. You 
will want to document both the process as well as the product (i.e., the design). 

Without adequate documentation it is difficult to: 

1. make the design, 

2. convince other people that the design is worthwhile or will work, 

3. explain what it is you plan or hope to do, 

4. recreate the design process used when it is necessary to determine why the 
design does not work, 

5. use a successful design process as the starting point for a new design, 

6. have someone else pickup and continue the project. 

Documentation, as a task, consumes much of an engineer's time. Documentation is 
not limited to design drawings but also includes written work, graphical work, computer 
codes, and oral presentations, as well as much of the effort spent on model 
development. V\lh\\e each of these areas requires its own special approach, the 
common thread is the desire to present the information in a form that is understandable 
to the reader. 

After a brief discussion of the differences between engineering and classroom 
documentation needs, this material discusses general documentation requirements, 
then addresses the documentation needs of modeling and finishes with some material 
related to preparing reports. The material should provide you with some general 
attitudes and strategies as well as some specific methods you can use. While there are 
no universally agreed-upon "correct" formats for documentation, the formats suggested 
here can be used as starting points and are acceptable for the work done in your 
design courses. 



11 The material presented here has been developed over the last seventeen years at Arizona State 
University in the course of teaching senior capstone classes, MAE 443, thermal system classes, MAE 
446, the principles of design classes, MAE 441, and the introduction to engineering class, ECE 106. The 

■ ideas and formats are a blend of the opinions and thoughts of the class faculty which include^rs. Neil 
Cooperrider, Bob Fries, David Laananen, Mark Henderson, E. Dan Hirleman, Barry McNeill, Jami Shah, 
Joe Davidson, Jim Blechschmidt, and Don Evans. 

I 
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Engineering vs. Homework Documentation 

It is not as if you have had no experience in documenting your work. During your years 
in high school and the engineering program you have /earned how to document (i.e., 
write up) homework problems and by the time you graduate you will have become quite 
proficient at homework documentation. Unfortunately, what you will soon discover after 
graduation is that the documentation effort which was adequate for homework problems 
is seldom adequate for engineering work. There are several reasons why more 
extensive documentation is needed for engineering problems. 

First, work and school environments are significantly different, particularly in terms of 
what information is known and need not be stated and what information is not known 
and must be stated. For example, in homework problems most of the information 
contained in the problem itself (i.e.. Problem Definition sort of material) need not be 
explicitly stated as part of the homework write up. Your instructor knows what problems 
she has assigned. Even material related to the method of modeling would generally not 
appear in your homework write up; the method is assumed to be some derivative of the 
material presented in the text or given in class. In an engineering work environment, 
however, you rarely have this built-in pre-knowledge about the problem and, unless it is 
provided in the documentation, it remains unknown to the reader. 

Second, the reasons for doing the work differ significantly in the two environments. In 
school the reason for doing the work is implicitly clear and need not be addressed, i.e., 
the reason for the work is to demonstrate the expertise needed to solve the problem. In 
an engineering environment the reasons for doing the work may or may not be clear. 
What is obvious to you may not be obvious to the reader. Unless you provide this 
information (i.e., context) in the documentation, the reasons for doing the work are lost. 

Finally, for homework problems there are generally no consequences or follow-up work 
required once an answer has been determined. The answer is boxed and turned in; 
just doing the problem is the end unto itself. This is clearly not the case for in an 
engineering environment. Merely doing the task is not sufficient unto itself. The results 
you obtain influence what you do next and this must be covered (explained) in the 
documentation. 

The sooner you recognize the differences between adequate homework documentation 
and adequate engineering problem documentation, the sooner you will start 
documenting your engineering design work well. 

Technical Work - General Comments 

Any work that is undertaken as part of the design process is classified as technical 
work. Work related to Problem Definition, Conceptual Design, "research", as well as 
Model Development and Use, all fall under the category of technical work. Buried, but 
hopefully not hidden, in this mass of technical work are the reasons why the design 
developed as it did. If the documentation effort is successful, then these reasons will 
not be hidden. This Section discusses general traits of good documentation and then 
addresses a number of specific documentation requirements. 
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traits of good documentation 

While specific methods of documentation vary depending on what sort of task you are 
documenting, there are some general, philosophical guidelines, which can help you 
achieve good documentation. When documenting your work, you should stnve to do 
the following: 

1. i mpart a sense of organization 

You should clearly show how the current work is related to the other tasks that 
are being or will be done; this ties the work together. You should strive to 
make it very clear where information comes from and where the reader can 
find information about other related tasks. It should be possible to quickly 
locate any desired task or piece of information contained within a task. For 
extensive work the use of page numbers and table and plot numbers helps add 
to an overall sense of organization. 

2. explain what is going on 

You should attempt to let the reader know, at all times, what is going on. This 
means explaining what is being attempted, why it is being attempted, and what 
method is going to be used. This also means that at the conclusion of a task 
you explain what you are going to do next as a consequence of the work just 
completed. 

3. make it readable 

Anything you can do to make the work easier to read will help improve the 
documentation. Imparting a sense of organization and explaining what is going 
on are two ways to improve readability. Other traits include doing neat work, 
having clean sketches, having the material bound so that it does not fall apart 
and yet is easy to access, etc. 

4. make it clear whose work it is 

it should always be clear who did the v/ork and when the work was done. This 
is usually done by dating and initialing each sheet of work done. 
You need to strive to incorporate all of the above guidelines into your documentation 
effort; failure to include one of the characteristics reduces your chance of having 
adequate documentation. 

graphical material presentation 
Much information is contained in pictures and hence your work will contain significant 
amounts of graphic information. This graphic material is generally either pictures of the 
design or plots showing the relationship between a performance variable and one or 
more design variables. Because pictures concisely contain so much information, i.e., 
pictures capture the essence of the material, graphic material is frequently copied and 
distributed, often without the accompanying text. Since this practice is so ubiquitous, it 
is critical, when preparing graphic material, that you 

1 . add enough annotation to the material that the picture makes sense standing 
alone, 

2. make the pictures with pencils or pens on paper that permits the making of 
good looking copies. 
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The first of these items means you put titles and labels on all your graphic material 
while the second item means that you do not put your drawings on green grid graph 
paper using a number 3 pencil. 

There are some sfanc/arc/s for the presentation of plots that you should be aware of. 

1 . Both axes must be labeled, including units when appropriate. 

2. The plot must have a descriptive title 

3. The dependent variable is always plotted on the vertical (y) axis. 

4. A concise legend must be added if the plot has more than one independent 
variable. 

5. Data points used in generating the plot are generally shown. 

6. Smooth curves are generally drawn through or near the data points to show 
the trend. 

The above standards are true for any plot, whether it is hand or computer drawn. You 
may find that computer drawn plots do not meet some of these annotation standards. 
In such cases it is entirely permissible to add the annotation after the plot has been 
generated. Whether you choose to add it by hand or with a typewriter or transfers 
depends on who is likely to see the work. The greater the chance that the work, or 
more likely a copy of the work, will be viewed by your supervisor or someone higher 
than your supervisor, the nicer you want the plot to look. 

Technical Work - The Modeling Tasks 

While the complete collection of technical work covers a number of different tasks, 
much of the technical work is concemed with developing and using models. Not only 
do the general requirements listed above apply to modeling, but there are some 
additional or more specific documentation requirements. The next three sections 
address the additional documentation needs for model development, computer models, 
and model use. 

who is the reader? 

How you document your technical work is heavily influenced by who is expected to be 
the reader. For these modeling tasks there are two potential readers and you must aim 
for the lowest common denominator when preparing the documentation. The primary 
reader will be you. You will need and want to refer to what you have done. The second 
reader will be a technically competent peer who may be expected to review the 
technical merits of the work or even be asked to take up the project and continue on 
from where you have stopped. 

documentation goals for analytical model development 

Model development work covers all aspects of assembling the set of equations which 
can be used to predict the performance of the design. The documentation of this task, 
i.e., the way you present the work, comes very close to how you have been doing 
homework preparation. After reading the model development material, if the material is 
well documented, the reader will have a sense that the work is complete and correct. 
This sense can be generated, if during the development, you are careful to: 



1. Define the Model Limitations, 

2. Define the Model Variables, 

3. Establish the Correctness of the Model. 

The first two items are requirements you've addressed in every homework assignment 
youVe ever had. You are continually reminded to list all assumptions and to show all 
your variables on a sketch. There should never be a question as to what a model 
variable represents. Defining the modeling variables includes defining the units 
associated with the variable. 

The third item is one that is often not explicitly part of your homework documentation. 
The correctness of the modeling method is generally implied and no real effort is 
needed to prove this. This is not the case for model development. As part of an 
engineering task you will need to impart a sense of correctness. There are several 
things you can do to demonstrate model correctness ranging from citing references, 
starting from a set of fundamental principles and logically developing the model, 
running a test case for which the answer is known, to proving that the assumptions 
used are reasonable. You may do all of these things. It is up to you to make sure that 
the reader comes away with a sense that the model and modeling method are correct. 
If you fail to do this, the reader will have grave doubts about the correctness of any 
decisions related to the use of the model. 

documentation goals for computer models 

The above discussion applies to any model that is developed, independent of the 
method or tools used in running the model. When developing computer models there 
are yet some additional documentation requirements which help assure that the 
computer model is understandable and correct. 

high level languages 

If the model is developed using a high level language such as FORTRAN or C, the 
computer code must include comments and a variable dictionary. A listing of the 
program must be included with the work. The listing should be in an appendix like 
location and should be easy to read (i.e., not bound into the documentation in such a 
manner as to make reading impossible). Any model which exceeds several hundred 
lines of code should be broken up into a set of smaller subroutines or procedures. 

formula solvers 

If the model is developed using a formula solving program such as TKIS0LVER2 then 
the documentation must include a completed Variable Sheet^ including the Units and 
Comments Columns. The Variable Sheet must show a consistent set of values for all 
variables. You must also supply a copy of the Rule (Equation) Sheet. The Rule Sheet 
should contain some general comments to tie major sets of equations together. If the 
model uses any special user defined functions these should also be included in the 
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2 TKISOLVER is a registered trademark of Universal Technical Systems, Inc., Rockford Illinois 

^ Equation solving programs have a variety of input and output screens which they call sheets. Some of 
these programs can solve multiple cases by running in what is called List Solving. 
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material submitted. User functions should be documented in the same fashion as the 
equations on the Rule Sheet. 
spreadsheets 

Documenting a model when a spreadsheet is used is a bit more difficult than when high 
level languages or formula solvers are used because spreadsheet equations use cell 
addresses rather than variable names. About the best thing you can do when 
spreadsheets are used to make sure each column and row of the sheet has a clearly 
defined title or heading. If the sheet's equations are not intuitively obvious then you will 
probably have to append a report which explains what calculation is being done in each 
cell. 

documenting the use of models 

Using a model implies running the model for a variety of different cases, in some 
instances a very large number of cases. It is easy to be ovenwhelmed by the enormity 
of data that can be generated with a model, and the documentation of this work needs 
careful consideration. 
displaying the results 

The important results of any case, such as those used to substantiate a conclusion, can 
be displayed using plots and or tables. Tables are required for any plot shown but there 
is no need to generate plots for all the tabular data, only the most important 
relationships need to be plotted. Each plot and table must have a figure or table 
number, e.g.. Figure 1 or Table IV. The data from all cases which are run using the 
same model {i.e., the equations are not changed from run to run; only the input data are 
changed) should be grouped together in chronological order. 

It is very important that for each case run, the values of the variables which were 
constant as well as the values of the variables which changed be known. The 
constants can be shown by including a complete Variable Sheet or table of values. For 
each case that you run you will generate a set of lists, probably collected into a table, 
showing how the other variables changed. For each case, you must annotate the lists 
(tables) with information explaining what has changed from the previous case. If you do 
use a general table with numerous variables, only some of which are of interest for 
each case, you must mark the variables of interest. This can be done using a 
highlighting marker. 

One last point, you should not include results from runs made but not discussed (i.e., 
not used). Filling a report or homework assignment with pages and pages of tables and 
plots, none of which are discussed (used), is not conducive to a good documentation 
effort. Work filled with such material, suggests an unorganized, haphazard approach to 
the problem, i.e., run a bunch of cases and hope something turns up. 

discussing the results 
When you present the results of your work you always want to discuss the results (i.e., 
you never just submit a plot or a table without some discussion of the table or plot). 
Look at the data and tell the reader what YOU, the writer, want the reader to notice; do 
not make the reader figure out what is important or interesting about the data. Explain 
what decisions you have made or will make based on the data. Try to explain why the 
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presented plot looks like it does (i.e., try to explain the fundaments.! laws and theories 
which are at work). If the data has any anomalies do not just ignore them; either 
attempt to explain why they are there or admit that after consideration you cannot really 
explain the unusual (unexpected) shape. The discussion must refer to the plots and 
tables by their names or numbers, e.g.. Figure 1 . 

Technical Work - An Entire Project 

The material up to this point has addressed the documentation requirements for 
individual models and not an entire project. The documentation for the entire project 
requires collecting and organizing all the individual tasks in one central location - the 
design notebook. 

the design notebook 

The design notebook is the complete record of the work done by your team on the 
project. The notebook is the source of all documentation required to establish that your 
design is feasible, that you considered a reasonable number of altematives, that 
system optimization was done, etc. It includes the early high level of abstraction 
concepts as well as the final low level of abstraction variables. 

design notebook organization 

There are two important guiding principles you want to keep in mind as you develop 
and create your notebook. 

1 . It must be easy to find work in the notebook. The page numbering scheme you 
use must make it easy to find work referenced to or to find material related to 
topics of interest (i.e., if suspension systems are part of the project, it must be 
easy to find the technical work related to suspension systems). 

2. It must be easy to insert new material into the notebook. By easy I mean that 
when pages are added to existing work you do not have to spend hours 
renumbering the notebook and revising all the internal references. 

There are several ways to accomplish these goals but most methods boil down to 
dividing the notebook up into a large set of technical sections which page numbers 
starting from 1 in each section. For example, if the third section concerns heat 
exchanger work, the tenth page of this work would be 3.10. You could have tasks 
within sections have sequential numbering (e.g.. A, B) and the work of each task start at 
page 1 . Thus for example in the previous case a page number of 3.B.2 would indicate 
that the heat exchanger work (i.e., 3) was in the second task (i.e., B) and on page 2 of 
that task, (note: organizing a notebook according to who has done the work is 
generally not a good practice, focus on the tasks not the people) 

the design section 

While each notebook will be unique in terms of what technical sections are present, 
each notebook will not be unique in the sense that all notebooks must have a design 
section. The design section of the notebook contains the latest version of the team's 
design. It will be an ever changing section, growing during the semester as more and 
more information about the design is known. An important point about the design 
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section: The design section of a notebook contains only the design (I.e., the 
description) and not any justification of the design. 

product and process 

Remember, one of the goals of technical documentation is to document the process as 
well as the product. This means that the notebook must do both and cannot be just a 
record of the current work or thinking. The only section of the notebook which is 
revised to show only current thinking is the design section. All other sections must 
clearly show process and product. Successful execution of this goal requires some 
thought about overall and section organization. 

The overall organization of the notebook generally includes: a Table of Contents up 
front, an Introduction to the project, the design section and all the technical sections. 
The organization of the material within each technical section needs to be such that it is 
easy for the notebook reader to tell what is going on, what the current state of the 
design process is and what are you currently working on (i.e., product and process). 

Reports of Technical Work 

While the documentation of your technical work is critical if you or anyone else is going 
to understand exactly what was done and why, tlk> people who generally make 
decisions about whether the project should be continued are not going to spend the 
time (do not have the time to spend) reading all this technical documentation. Instead 
you must prepare reports which serve the purpose of summarizing all your work and 
presenting the results in a format which is persuasive and able to convince a reader of 
the merits of the work. The ability to write and/or give good reports is critical if you want 
people to respond favorably to your efforts. 

There are a vast variety of report types, ranging from short memo-type progress 
reports, up to extensive multi-volume reports. Many of the required reports are oral, 
with only limited written material to accompany the presentations. There are progress 
reports, conceptual design reports, final design reports, and documents (proposals) 
requesting money. At some time in your career you will have to prepare all or portions 
of ail of these reports. 

before writing your report 
Good reports do not just spring into existence; good reports take a conscious effort. 
Writing a report is a process just as creating a design is a process and there is no one, 
correct v/asj to do this process. Each person eventually determines which process best 
suits them' But as with the design process there are some guidelines which will help 
you learn to develop a report writing process you feel comfortable with. Following are 
some general tips which should ameliorate learning this process. 

know your audience 

You must determine who you are writing the report for. Is it for your techiiical peers, 
your parents, your technically wise supervisor, the vice-president of research, the 
nightly news, your prime contractor, a sub-contractor, a vendor, etc. As is the case in 
design you must know who your customer is, i.e., who is going to read the material or 
listen to the talk. 
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Once you know your customer you then know what level of detail must be included in 
the report. You know what vocabulary you can use. You know what ideas need 
explanation and what ideas the reader already understands. You don't want to waste 
the reader's time by telling her stuff she already knows but you also don't want to loose 
the reader because they haven't a clue as to what is going on. 

know your purpose 

You must know explicitly the reason for the report. What do you hope will happen after 
the customer has read the report? When you know this then you know exactly what 
things to put in the report (things which will make the desired result happen) and what 
things to leave out (things which either have no bearing on the desired outcome or 
worse yet will cause the customer to not perform the desired outcome). 

design briefs 

Design briefs, as the name implies, are relatively short documents which are meant to 
initiate some work. While a brief can be written during every stage of the design 
process, all design briefs contain three common elements. The brief must contain the 
goal(s), constraints, and measures of merit (quality, goodness) which are appropriate 
for the task at hand. Writing a good brief takes some effort; it is a balancing act. The 
brief must be as detailed and specific as possible so that the work and potential solution 
match your desires while at the same time the brief must not be too restrictive (i.e., 
implied solutions, solutions which you are not wedded to) or you run the risk of unduly 
biasing or limiting the person who is to do the work. Within this balancing act the brief 
should be as quantitative as possible (e.g., last at least five years rather than have a 
long life if in fact you really desire a five year life). You should try to clearly define 
concepts which may have alternate or multiple meanings (e.g., low specific fuel 
consumption or time to process rather than high efficiency). A well written design brief 
would be one where you gave it to the person to do the work, left for a month, and 
came back to find the task done to your satisfaction (i.e., no surprises). 

design proposal 

A Design Proposal presents a proposed solution (i.e., a possible concept) along with 
details of how the design will be developed. The purpose of a Design Proposal is to 
convince decision makers, i.e., the people who are responsible for deciding whether the 
project should proceed, that you understand the nature of the problem and know how to 
solve it. It is often the only document read by these decision makers and thus it must 
be very persuasive. If the project proceeds, the Design Proposal becomes a contract, 
defining what will be done and what will be supplied when the work is finished. 
Depending on who the decision maker is (i.e., is he/she your immediate supervisor or a 
person outside your company). Design Proposals can range from short memos to 
extensive reports. A written Design Proposal should always be developed, even if the 
initial approval is the result of conversations or a meeting. 

A Design Proposal can only be written after some significant prelimina'y work has been 
done. Only after the problem has been defined, only after significant numbers of high 
level of abstraction design altematives have been developed and evaluated, and only 
after a detailed plan of attack (i.e., a detailed design strategy) has been developed is it 
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possible to write a Design Proposal. Only then can you expect to be able to develop a 
document which will be convincing. 

Without a significant effort spent on problem definition, you cannot expect to convince 
the reader that you are working on the correct problem and that you are considering all 
the important, and not just the "obvious," aspects to the problem. Without a serious 
effort spent in conceptual design, you cannot expect to propose a solution that seems 
promising. Without quantitative based decisions for selection of the proposed 
alternative you cannot expect the reader to appreciate your technical skills. Arid finally, 
without a well defined design process you cannot expect to convince the decision 
makers that you can successfully accomplish the complex task. 

design proposal format - senior design projects 
There are many formats possible for a Design Proposal and you will need to determine 
the correcf format. The following format is the one to be used for the senior design 
project classes (i.e., MAE 443/468). The proposal has several required major written 
sections which are discussed below. 

1 . Title Page 

Each Design Proposal must have a Title Page. This page must include the 
project title, the team number, the date the Design Proposal is submitted, the 
team members' names, and the names of the course faculty. Leave a blank, 
signature line to the right of each name. 

2. Introduction 

This material sets the context and helps to define the boundaries of the 
problem. Who is the customer; why is your team working on this project? The 
context portion of this material will require a little imagination on your part. 
Since this problem is an assigned class problem there is no clearly defined 
context, i.e., work environn.ent. It is up to the team to define a context (i.e., 
work environment within which the proposed project makes sense). Defining 
this context helps define the boundaries of the problem; it establishes what 
things will be fixed and cannot change, e.g., environmental and system 
parameters, and what things can and will be varied, e.g., design and system 
variables. The team must define how their work fits into the larger problem 
solving process. 

3. Problem Definition 

This material defines and explains the design objectives and all the relevant 
performance requirements, other constraints, and goals (i.e., discuss all the 
factors which must be considered in establishing design feasibility and quality). 
Any known restrictions on design variables must be explained. This section of 
the report will be used by the course faculty in deciding whether the team is 
planing to consider all the important performance and failure mechanisms in 
their up coming work. The Design Proposal will not be signed until this section 
is deemed reasonable. 

4. Concept Generation and Selection 

This section needs to give a sense of what alternative concepts were 
considered and how these altematives were winnowed down to the final 
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concept to be developed during the semester. All the important, high level of 
abstraction alternatives should be described. The decision process should be, 
as much as possible, based on sound technical (i.e., read hard quantitative) 
reasons rather than intuitive (i.e., read soft qualitative) reasons. It is not 
necessary to develop (i.e., describe) all the alternatives in equal detail; it is 
important that the concept of choice be fully, at the level of abstraction 
appropriate, described. 

5. Design Process 

This can be a very important section of the proposal. This section can convey 
mastery of the design strategy generation and show how the team might 
monitor and control the design process. It is here that the global approach is 
explained. A context for the work has been defined in an eartier section as has 
a possible design. The design process has proceeded to the point where the 
design contains some generic components, defined by some initial geometric 
and materials information for the various components Given this scenario and 
assuming that each team member will be assigned a certain set of the 
components to work on, this section describes the design process to be used 
by your team to work on this problem. Develop the process in enough detail 
that each member of your team understands her role within both the team and 
the company (context consistency) and knows what the appropriate set of 
design variables would be. Try to use the terminology used throughout your 
system design class (e.g., design variables, system parameters, system 
variables, components, design space, etc.) in defining and describing the 
proposed design process. Feel free to define the role of the team within the 
entire project; just make sure this role is clearly explained to your team 
members and that the role is consistent with the general role outlined in the 
Introduction. 

This section lets the team define the system aspects of the proposed artifact 
(solution). The specific technical models and methods are defined in the 
Statement of Work section, assuming this section is part of the report. In this 
section a picture i" painted showing how all the work blends together into an 
organized coherent approach. 

6. Statement of Work 

It is here that the technical details of the proposed plan of attack are given and 
discussed. Based on how the team has broken the problem down there will be 
a number of components that must be developed where the development will 
require the execution of a variety of tasks (e.g., predict component/system 
performance). This section of the report defines all these tasks which must be 
done. The more detail that is included in this section , i.e., the more tasks that 
are included, the better the chances are of convincing the reader that the team 
has a complete, technically sound plan. For example, in the design of an 
airplane, a team could include a task called "Design of the Wing;" or they could 
rather break the task down into a number of tasks ( e.g., "Aerodynamic Wing 
Loads", "Wing Material Selection", "Wing Structural Stresses", etc.). 

Each proposed task should be a separate paragraph and should address: 
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1 . the goal or purpose of the task, 

2. how the task will be done, i.e., describe what modelling techniques will be 
used, what generic models will be used (e.g., Bernoulli, FEM), what 
resources used, etc., 

3. who is going to do the work. 
7. Deliverables 

This is the section of the report which is primarily used to judge whether the 
proposed level of effort is reasonable. This material describes what things are 
going to be turned in (i.e., delivered). There should be an item for everything 
that is turned in during the semester, including Notebook reviews, outlines, etc. 
The date that each of these items is delivered should be shown on the 
Schedule (see next section) as a milestone. Except for the final design 
specification contained in the final Notebook submittal, there needs only be a 
very briefly description of what is being delivered (i.e., the name of the 
document and a sentence or two is sufficient). 

The exception to this brevity is the description of the detail contained in the 
Design Section of your last Notebook. This report must describe, as best 
possible, the level of detail to be contained in the final design the team 
eventually plans to submit. List as many of the design variables as possible 
Gust their names not their values). This large set of variables will need to be 
organized is some fashion (e.g., by component or function or system). Try to 
indicate the number of significant figures that will be presented. If the design 
variable value is something other than a number, indicate whether the variable 
value will be generic or specific (e.g., if the design variable is a type of pump, 
will its value be "gear pump" (generic) or Goulds Pumps, number G23-a45 
(specific)). In addition to listing the design variables this section must also 
describe what set of design drawings (i.e., assembly, isometric with cutaways, 
layout, etc.) the team plan to develop and submit. Try to indicate what parts of 
these drawings will have dimensional information explicitly shown. 

8. Schedule 

Because there are several mastery areas related to schedules (C4 & G2) the 
report must have a schedule showing when all the proposed tasks will be done. 
Any format is possible; a Gantt Chart format is acceptable (dates across the 
top, tasks down along the side, and bars showing when each task starts and 
ends). If the report has a Statement of Work Section there should be a one to 
one correspondence between the tasks shown on this schedule and the tasks 
discussed in the Statement of Work. In the Gantt Chart the thickness of the 
bars should indicate the level of effort (i.e., how many team members 
assigned), thicker bars implying more effort. The schedule should also include 
all required milestones (e.g., design notebook due for review) 

final design report 

The purpose of a final design report is to document and justify the final design. As is 
true for any report, this report must be convincing. Some general traits of good 
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documentation were given earlier; some more specific suggestions for this report are 
discussed below. 

1. present numerical results 

The use of a specific numerical value or result instead of a general term always 
implies a greater level of known detail in the work. For example, stating that 
the temperature was 120 C, rather than that the temperature was high makes it 
much easier to convince the reader that some decision, based on this 
temperature, is reasonable. The presence of numbers implies the existence of 
models which again adds credibility to the report. 

2. present specific facts 

This is very similar to the above item but is not limited to numerical data. The 
more specific you can be, the stronger your arguments become. For example, 
the reader gets a much better feeling when he/she reads "The critical column 
load was found using a modified Euler, slender column, buckling analysis;" 
rather than, "The maximum column load was determined. 

3. present a feeling of completeness 

If the reader believes that yci have covered all the necessary topics he/she will 
be more receptive to accept the work and results presented. This sense of 
completeness can be fostered by, early on, listing what you feel are the 
important topics and then discussing each of them completely in the body of 
the report. This is standard good report writing, i.e., tell t he reader what you 
are going to discuss and then discuss the items. 

4. present an organized feeling 

The more organized your report; the more organized your project or at least 
that is the implication. If conclusions arise naturally as a conseguence of a 
series of arguments, the reader gets the feeling that the writer of the report has 
truly mastered a complex subject because he/she is able to present the myriad 
of facts in a meaningful, organized manner. 

5. present a readable report 

If a report is easy to read, the reader is likely to accept the arguments 
presented more easily than if the report is hard to re ad. Report readability can 
be enhanced by using transition paragraphs to tie major sections of the report 



The need to be readable and the desire to use specific numerical facts can 
lead to a conflict. The presence of too much numerical data and too little text 
can make the reading very dry ("wooden" as Zonker remarked on the style of 
the Zip Code directory he was reading) and not very readable. You have to 
walk (write) a fine line between presenting too much or too little numerical 
work. 

define your audience 

For your final report you may select your audience to be either the course instructors or 
the MAE faculty. You must make it clear to the course instructors which audience you 
have selected. 



together. 
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state your purpose - select a theme 

At the time you tel! the course instructors who your audience is you must also tell them 
what your theme is. The quality and completeness of this final report will be judged 
based on your success at achieving your state purpose or theme. 

final design report format - senior design projects 

The following format is the one to be used for the senior design project classes (i.e., 
MAE 443/468). The report has six major written sections: Executive Summary, 
Introduction, The Design Process, The Design, Design Justification, and Conclusions. 
Each of these, along with some other topics, is discussed in the material to follow. As 
you write each section, be sure to keep the guidelines for good report writing in mind. 

Before writing the report you should develop an outline. Since the report is limited in 
size and there are many topics which must be covered, an outline is almost mandatory. 
Further, since the report will have material written by different people, the outline is one 
way to define exactly what material should be covered by each person. An outline 
helps organize the report and eliminate redundancies. 

1. title page 

The Title Page is the first page of the report and must include the project title, 
the class name, the submittal date, the group number, the members' names, 
and the instructors' names. 

2. design picture 

This page is your concept presentation page. This page is a picture, sketch, or 
set of pictures on the page, that shows the essence of your artifact. The more 
you capture the sense of the entire proposed artifact the better this page 
becomes. It can range from an artists rendition to an engineering drawing; 
however, the more it is a rendering the more successful the picture will be. Be 
creative in developing this picture. 

3. executive summan/ 

The Executive Summary is a summary of the report. This summary must 
included introductory background material as well as a synopsis of the design. 
In addition, you must cover the reasons behind several of the more important 
design decisions. The Executive Summary, along with the design Picture must 
be able to stand alone. 

Note: the title page, design picture, and executive summary should form a 
package which could stand along (i.e., make sense with no other material). 
These three pieces of a report are often removed, copied and distributed to other 
people. This means that the audience for the executive summary is larger and 
potentially less informed than the audience for the report itself. 

4. table of contents 

Following the Executive Summary comes the Table of Contents. All major 
sections of the report should appear here, including the Executive Summary 
and al! Appendices. You should have a separate Figure Table of Contents 
following the first Table of Contents. Every Figure (i.e., every plot and table) 
must be entered with a page number. 
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5. introduction 

This material is very similar to the material contained in the Problem Definition 
Section of your Design Proposai This section establishes the context of the 
problem as well as presenting a brief description of the concept you propose as 
a solution. The discussion of the concept is done in a general manner, 
explaining how the concept works and solves the stated problem. The 
discussion of the concept does not contain specific details about the particular 
artifact you eventually designed; this detail will be found in the Design Section 
of this report. 

This section gives reasons as to why the project is being worked on; it gives 
background material or technical information which gets the reader up to 
speed; and it contains material which establishes the important design 
constraints and selection criterion. This section could contain justification for 
the general concept (some sort of shortened version of what was in the design 
proposal) but should not contain justification of the design decisions you made 
after the Design Proposal was signed. 

This materia! may end with a transition paragraph which explains the 
organization of the rest of the report. 

6. the design process 

In your Design Proposal you discussed the general approach you planned to 
use, i.e., you discussed the system aspects of your work. In this section of 
your report you should explain what general approach you actually used. 
Discuss the generic types of models that were developed and how they were 
used and integrated together. Specific details conceming the models are not 
of interest; how tfie models were used in the over all design process is of 
interest. 

7. the design 

This section presents your final design, the best design you had time to find. 
This section summarizes the most important features of the design. This 
section is not meant to be a complete description of the design because that is 
found in the Notebook. This section will be a combination of text, tables, and 
drawings, organized and presented in a form that makes it easy for the reader 
to quickly grasp the important features and components of the proposed 
artifact. As is true with the Notebook design section, this section of the report 
should not contain any justification of the design. All justification for the 
selections will be found in the next section. 

8. d esign justification 

This section will be the longest and most detailed section of the report. It is in 
this section that you attempt to convince the reader that, through sound 
technical work: 

1 . the design is feasible and 

2. the design is good. 
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You need to present plots and tables which show that the design does not 
violate major constraints (i.e., it is feasible) and that the particular values of the 
design variables selected give a design which is better, according to the 
selection criterion outlined in the Introduction . This section can also be used to 
discuss the sensitivity of some of the environmental and performance variables 
on the design decisions made (e.g., how sensitive is the wing lift calculation to 
the value of air density used in the analysis). If your group has done redesign 
this is the place this worl< is discussed. 

Be sure to completely discuss all the figures that you include as proof of design 
feasibility or goodness. Be sure to discuss the implications of the plots and do 
not simply report the shape of the plot. Try to explain why the plot looks like it 
does. This may require you to briefly discuss some of the modelling details 
(e.g., major assumptions, first principle approaches used, etc.). Discussion of 
specific model development should be kept to a minimum and only be 
presented if it helps explain the predicted results. You should emphasize how 
the results of the modelling, as shown in the figures, influenced your design 
decisions. The detailed information on how the tables and plots were 
generated is contained in the Notebook. 

9. conclusions 

This section ends the body of the report and lists the major conclusions 
reached concerning the design. It also gives any recommendations you may 
have reached concerning the need for additional work, the goodness of the 
design or the completeness of the modelling. This is an important, if not long, 
part of the report. 

10. appendices 

These are optional and contain information which you feel should accompany 
the report. Remember, however, the report must make sense without the 
Appendices. 

11. general format notes 

You should assemble your report in the following order, observing the page 
limits. 

1 . Title Page - one page 

2. Design Picture - one page 

3. Executive Summary - two page maximum 

4. Table of Contents - no limit 

5. Body of Report - twenty pages maximum (exclusive of pictures, plots, tables, 
etc.) 

6. Appendixes - no limit 

You may use any printer font you wish but it must not be smaller than 12 point. 
The report should have at least a one inch margin on all sides. The report 
should be stapled once in the upper left hand corner; no folders or binders are 
required or desired. 
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oral reports 

Dog and Pony Shows, Informal Presentations - whatever you call them -- oral 
presentations of your work are an important facet of your professional life. Lasting 
Impressions are made during oral presentations, impressions which can be helpful or 
harmful. The long range implications are such that you should never just give one of 
these presentation "off the top of your head " (i.e., you must plan in advance what you 
want to cover and how to cover it). 

Like all aspects of design, the only way to learn how to present good oral presentations 
is to give them. It takes practice. Here are a few do's and don'ts which may help you 
initially. 

1. plan your talk 

Find out who the audience will be and how much time you will have, and plan 
your talk around those two constraints. If the audience is familiar with your 
work, you will not need much introductory material and vice versa. Make an 
ordered list of the topics you want to cover and then, starting with the first one, 
cover as many topics as you think you will have time for. Always cover the 
most important topics first. 

2. use visual aids 

It is mandaton; that you use visual aids, either overhead slides, 35 mm slides, 
video, etc. You can use the slides to estimate the length of your talk. On the 
average you can cover 1 to 2 slides per minute of presentation. It's possible to 
go through 45 slides in 15 minutes, but no one will understand what is on any 
of the slides and the presentation will have failed. Your slides must be kept 
simple. Do not attempt to put all the information possible on one slide, either 
remove the secondary information or present 2+ slides on the subject. 

3. present report extemporaneously 

You should strive to present the material with few if any notes, using the slides 
to organize the talk and remind you of the key items you want to discuss. 
"Extemporaneous" does not mean "off the top of your head," it simply means 
that you are not reading a prepared script. If you gave the talk twice, each talk 
would be a little diffe .- 'it. You should never just read the text of a report. 

4. practice 

Especially when you are just beginning to give oral presentations, you need to 
pre. -^ice delivering the talk. This helps you see if the timing is correct as well as 
developing a presentation pattern in your mind, a pattern you can 
subconsciously use when actually presenting the material. 

5. present to the audience 

When giving the talk, look at the audience; this way you can judge the 
reception of the talk. It is fine to glance at the slides from time to time, but you 
should not spend you whole time talking to the screen or slide projector. (Very 
few projects have been funded by slide projectors.) 
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Expert-Novice Differences in Problem Solving 

(From Mestre, 1991) 



The following three physics problems can be used to illustrate differences in the problem solving 
behavior of experts and novices: 
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Problem 1 



A 1 kilogram stick of length 1 meter is placed on a frictionless 
horizontal surface and is free to rotate about a vertical axle 
through one end. A 50 gram lump of clay is attached 80 
centimeters from the pivot. Find the net force between the 
stock and the clay when the angular velocity of the system is 3 
radians per second. 




Problem 2: 



A stick of length 1 .5 meters and mass 0.2 kilograms is on a 
frictionless horizontal surface and is rotating about a pivot at 
one end with an angular velocity of 5 radians per second. A 35 
gram lump of clay drops vertically onto the stick at its midpoint. 
If the clay remains attached to the stick, find the final angular 
velocity of the stick-clay system. 




Problem 3: 



A 60 kilogram block is held in place in a frictionless inclined 
plan of angle 30 degrees. The block is attached to a 
hanging mass by a massless string over a frictionless 
pulley. Find the value of the hanging mass so that the 
block does not move when released. 




Question: Which of problems 2 and 3 would be solved mosi like problem 1? 
Explain your answer. 



Typical Expert's Response: 
Typical Novice's Response: 



Problem 3 would be solved most like problem 1 because 
both involve the application of Newton's Second Law. 

Problem 2 would be solved most like problem 1 because 
both involve a rotating stick with a lump of clay attached. 



Note that the expert cues on the underlying principle that could be applied to solve the 
problems, whereas the novice cues on the surface characteristics of the problems. 



Question: Describe how you would go about solving problem 1 . 

Typical Expert's Response: The clay accelerates as it moves in a circular path. 

The net force needed to keep the clay going in a 
circle is provided by the horizontal force between the 
stick and the clay. Therefore, apply Newton's Second 
Law and set the net force on the clay equal to its 
mass times its centripetal acceleration. Then solve 
for the magnitude of the force. 



Typical Novice's Response: The stick and the clay are both moving in a circular 
path so I would probably have to use /co and -^/co^ for the stick, and mvR and ~/nv^ for 

the clay. I am told values for the mass of the clay and the stick so I have m and I can 
find / by looking up the moment of inertia of a stick pivoted at one end in a table and 

plugging in to get a number for it. The force for something moving in a circle is — so I 

R 

think that I have enough to get an answer. 



Note that the expert performs a qualitative analysis during which the applicable principle 
is identified and a procedure for applying the principle is stated. In contrast, the novice 
immediately resorts to formulaic approaches, often writing down expressions that are 

irrelevant for solving the problem (e.g., /co, -/co^). Principles and concepts are usually 
lacking from the novice's approach. 
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One approach for helping students overcome the "force of the hand" misconception 

(from Mestre, 1991) 



Probe for Misconception: Toss a ball vertically up and ask students to enumerate the 
forces acting on it when the object is halfway to the top of its trajectory. 

Ask questions to clarify students' beliefs: Does the "force of the hand" change in 
magnitude or direction? What happens to this force at the top of the trajectory and on the 
way down? Is this force active in other situations, such as rolling a ball on top of a horizontal 
surface? When does the "force of the hand" act on the ball? 

Suggest discrepant events that contradict students' beliefs: Suppose I push on you - 
how do you know when I stop pushing on you? How does the object "know" that the "force 
of the hand" is still acting on it? If the object experiences the "force of the hand" after it 
leaves the hand, why can't one control this force while the ball is in the air? 

Encourage discussion and debate: Promote fruitful, non-disparaging debate among 
students as they take different slides in the ensuing argument. Encourage students to apply 
physics arguments, concepts, and definitions. 

Guide students toward constructing scientific concepts: How one guides students 
depends on their awareness to the teacher's questions and the issues raised during the 
discussion and debate. Once could involve students in: 

• A synthesis of their responses to questions and situations, with a discussion of 
how consistent those responses are with the scientific concept or other 
observations. 

• A discussion of "thought experiments" that in principle could measure the 
"force of the hand." 

• A discussion of what the motion would be like with and without the "force of the 
hand" from the perspective of Newton's Second Law. 

• The design and execution of experiments to test hypotheses. 

Reevaluate students' understanding: Ask questions and pose situations that allow 
students to display whether or not they have acquired the appropriate understanding: 

• When is the "force of the hand" acting on a ball that is thrown up in the air? 

• What are the forces acting on a cannonball that was shot out of a cannon while 
it is airborne? 

• What is the difference, if any, between the cannonball and the thrown ball? 



A classroom dialogue for clarifying the concept of acceleration 

(From Mestre, 1991) 



The teacher has previously introduced the concept of acceleration. The 
teacher now presents some simple situations in order to explore the 
students' understanding of the concept in concrete contexts. 

Teacher Suppose I toss a ball straight up in the air like this 

(demonstrates). What is the ball's acceleration at the top of the 
trajectory? 

Student 1 : Zero. 

Student 2 : Yeah, zero. 

Teacher : Why is it zero? 

Student 1 : Well, at the top the ball stops moving, so the acceleration must 
be zero. 

Teacher OK. If I place the ball on the table so that it doesn't move, is it 
accelerating? 

Student 2 : No. It's not moving. 

Teacher : What if I roll the ball across the table so that it moves at a 

constant velocity (demonstrates). Is the ball accelerating in that 
case? 

Students 1 

and 2 : Yeah. 

Student 3 : No way! If the ball is rolling at a constant speed it doesn't have 
any acceleration because its speed doesn*t change. 

Student 2 : No ... listen. The ball had to have an acceleration to get to the 
speed it had. 

Student 3 : Yeah, but once it roils at a constant speed it can't have any 
acceleration, 'cause if it did it would roll faster and faster. 

Student 2 : I'm not sure. You're confusing me. 

Teacher : What's the definition of acceleration? 

Student 1 : It's the change in speed over the change in time. 
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Teacher : Close but not quite. It is the change in velocity over the change in 
time. Speed doesn't care about direction but velocity does. At 
any rate, apply your definition to the ball rolling on the table. 

Student 2 : Well, I guess since its speed - 1 mean, velocity - doesn't change 
when it rolls; it can't have an acceleration. 

Teacher Do we agree on this case? 

Student 1 : Yeah. 

Student 2 : 1 guess so. 

Teacher : So it appears that an object can have a zero acceleration if it is 
standing still or if it is moving at a constant velocity. Let's 
reconsider the case where the ball is at the top of its trajectory 
(demonstrates again). What is the ball's acceleration when it is at 
the top? 

Student 3 : It would be zero because the ball is standing still at the top. it's 
not moving • it has tot urn around. 

Student 2 : It think it might be accelerating because it gets going faster and 
faster. 

Student 1 : Yeah, but that doesn't happen until it gets going again. When it's 
standing still it's not accelerating. 



The teacher could pursue various directions from here to attempt to get students 
to realize that the ball's acceleration is not zero at the top of the trajectory. One 
might be to pose a related situation. Another might be to revisit the definition of 
acceleration and ask students to apply it during the time interval just prior to the 
ball's reaching the top and just after the ball starts its descent. 
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Ernst von Glaserfeld on Constructivism 



"...[constructivism] deliberately discards the notion that knowledge could or 
should be a representation of an observer-independent world-in-itself and 
replaces it with the demand that the conceptual constructs we call knowledge be 
viable in the experiential world of the knowing subject." (Synthese, 1989, p. 122) 

"...knowledge cannot simply be transferred by means of words. Verbally 
explaining a problem does not lead to understanding, unless the concepts the 
listener has associated with the linguistic components of the explanation are 
compatible with those the explainer has in mind. Hence it is essential that the 
teacher have an adequate model of the conceptual network within which the 
student assimilates what he or she is being told. Without such a model as basis, 
teaching is likely to remain a hit-or-miss affair." (Synthese, 1989, p. 136) 

"...the fact that scientific knowledge enables us to cope does not justify the belief 
that scientific knowledge provides a picture of the world that corresponds to an 
absolute reality." (Synthese, 1989, p. 135) 

"...if I want to 'orient' the conceptual construction of others, I would do well to 
build up some idea as to what goes on in their heads. In other words, in order to 
teach, one must construct models of those "others' who happen to be the 
students. Only by operating on the basis of a more or less adequate model of 
the students' conceptual structures can one present the required 'knowledge' in 
ways that are accessible to the students. And students obviously do not come 
as blank slates. They have their own constructs, as well as theories of how and 
why their constructs work. Such constructs or theories may be considered 
'misconceptions' from the teacher's point of view, because they are incompatible 
with the concepts and theories sanctioned by the particular discipline at the 
moment. Nevertheless they make good sense to the students, precisely 
because they have worked quite well in the context of the students' interests and 
activities. And because these concepts and theories make sense to the 
students, they also determine to a large extend what the students see. Hence it 
is often necessary to do a certain amount of dismantling before the building up 
can begin." (Bremen Proceedings, 1992) 
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Commentary on Hands-On Activities 



"Unfortunately, the research evidence suggests 
that hands-on activities or instruction in process 
skills will not ensure meaningful learning, either 
alone or in combination with conventional fact- 
based instruction. One problem is that science 
processes do not seem to consist of unitary 
skills that can be transferred from one context to 
another. Observing cell cultures, for example, 
has little in common with observing geological 
formations or with observing chemical reactions. 
Furthermore, a major component of process 
skills seems to be content knowledge (e.g., a 
good observer of cell cultures must know a lot 
about cells.)" (Anderson, 1987) 
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Incorporating Assessment Into Classroom Activities 



Richard S. Culver 
SUNY-Bmchamion 



*A is for aiscismcnl Asscssmem b the most powerful tool you have for influoKini the 
learning process. Change the assessmens and you change what smdents give their atlennon 
to. Keep control of the assessment and you keep control of kaming. \ If you want students 
to take some responsibility for their learning, then you probably have to hand over aspects of 

assessment toa 'I I) ^ 

. — Graham Gibbs 



INTRODUCTION 

Gibbs' quoic reflects an often overlooked truism in 
modem education - Grades are the coin of the realm in 
the classroom. For most students, assessment is synony 
mous with grades. But assessment should accomplish 
more than just providing a basis for assigning grades. It 
should also provide feedback to improve performance by 
the student aod course effectiveness by the instruaor. 

There is an interesting parallel between the 
•Quality" movement in industry and the role of assessment 
in education. The marked improvements in production 
efSciency and product quality achieved in the past five 
yean have resulted primarily from a change in manage- 
ment pbilosopby. Responsibility for product quality has 
been shifted to the workers, along with mechanisms for 
obtaining direct feedback on the production process. 
Through quality circles, workers have joined the team 
which manager the production process, contributing to 
soluticris of problems and new innovations* 

The comxnlional approach to engineering educa- 
tion is similar to traditional management phDosophy. 
Students, like workers on the assembly line, are assigned 
tasks to complete, including homework assignments, lab 
reports, and examinations. The instrunor grades these to 
determine the qoiality of student perlorroancc and provides 
& numerical cvaJuation of the result back to the student 
The primary purpose for the evaluation is to assign a 
grade. The stucicnt is given little opportunity to provide 
feedback on tbc educational process. The end-of-term 
student course evaluation can be accepted or rejened as a 
basis for future course/instructor improvement For 
meaningful improvement in the quality of education, we 
need to design much more sophisticated and elective 
classroom assessment systems. In this paper, i model of 
incorporating a broad-based assessment program into a 
course is presented. 



K, Patricia Cross, the 1991 Distinguisbed Lecturer, 
states, *We will not make effective progress in reducing the 
gap between what is taught and what is Icaroed until the 
classroom instructor is more actively involved in the 
assessment process.*(2] She distinguishes between 
pc^p<L<;ment-fnr>accountabilitY. in which the feedback is 
public, normative, comparative, and competitive, and 
^^<^g<L<;ment«for-improvemenL which provides a continuous 
flow of information to aid in shaping teaching and learning 
while in process, Asscssmect-for-improvemcnl, which 
parallels the industrial 'quality* philosophy, is typically 
formative, b most effective if bqi made pubbc and 
emphasizes comp<*tencies rather than comparisons, 

COURSE OBJECTIVES AND ASSESSMENT 

As in industry, quality must be defined in terms of 
outcome specifications. Without performance objectives, 
which define what the student should know and be able to 
do. it is difficult to design an effective assessment system- 
Professor John Heywood, author of Awssnvgnt in Hipher 
pf1uc3tion[ 3]. describes efforts to improve assessment in 
English universities in this way. If learning b to be 
enhanced, then the design of comprehensive examinations 
becomes a complex activity which has to takjc into account 
the effects of instruction on learning and the attainment of 
objectivcs."(4) He illustrated various compoDcnu in the 
design of curriculum in the cycle shown in Figure 1. It 
should be noted that, while the syllabus drives the learning 
strategies, the syllabus itself works in interacrion with aims, 
student assessment, program evaluation, and materials. 

In response to Hcywoods premise, I have found it 
helpful to write three independent, but complemeniary, 
sets of objectives for a course.which define ^iiat the 
student should knovk' Qmowi^dge^ . be able to do (siills). 
and how he should feel about it (attitudes) . The 
obieciivcs form a living which will change as the course 
develops and in response to the personality of each class. 



1992 ASEE Annual Conference Proceedlngi 

790 



3G1 



BEST COPY AVAILABLE 



I 
I 
I 
I 
I 
I 
I 
I 
I 
i 
I 
I 
I 
I 
I 
I 
I 
I 



DO 



CXAMlHATIONt 
4 A(SESSMEKT 




TEST 



EVAtUATK>H Of WMOU COUR« 

Figure 1 • Assessmenl-Curriculura-lnslruclioD Process 

For the objectives to be of value, thc%- musl be 
used before, during, and after the course as pan of an 
ongoing evaluation of student learning effecijveness. 
Students should be provided with a copy of the objectives 
at the beginning of the term and have their irjponance 
emphasized. The objectives will define the lc%-el of mas- 
icry required in each topic and suggest methods of evalua- 
tion which will determine if the objectives harvc been meL 

KOLB'S EXPERIENTIAL LE>IRNING CYOUE APPLIED 
TO A TECHNICAL CONCEPT 

In previous papers, I have described the stages of 
^l^ellcctual development of college students ais defined by 
JbrTy.[5) It is assumed that such developmem is desirable 
'^d that properly designed educational programs can stim- 
ulate intellectual grovrth. in contrast to the Lack of grovrth 
exhibited by students in conventional progracis. As 
previously mentioned, assessment is perhaps the most 
powerful tool for encouraging students to sueich beyond 
their current level. For growth to occur, the situdents must 
be stretched in a manner which is not always comfortable, 
to reach for a goal that they think is beyond ibem. In 
conventional courses, intelleauai challenge is almost 
entirely based on increasingly difficult analysis of more 
advanced material, but not synthesis or evaluaiiotL It does 
not provide alternative methods of learning tbe material or 
for demonstrating that knowledge. Combining Perry's 
scheme with Kolb's Experiential Learning C>cle, along 
with the assessment methods which are available at each 
suge, provides a model for building mastery of a subjea 
while stimulating intelleauai development 

The Learning Cycle - David Kolb describes a logical 
sequence of events through which an individual will pass in 
learning a new concept: concrete experience, reflective 
observation, abstract conceptualization, and active 
cxperimentation.[6] As shown in Figure 2, ibcse can be 
descri;>ed for learning in a technical course in terras of the 
anion verbs: Do, Think, Model, and TesL 




Actlvt 

£ip«rlmentiitlon 



Abstrvct 
Conceptuaiftzjtk>n 



Figure 2 - Ko!b*s ExperimentiaJ Learning Cycle 

The Currlculura Cycle - Tbe first step in designing a 
course to teach a new concept is to write a scries cf 
learning objeaives. Next, a scries of aaivities which 
follow Kolb's suges of learning are selected that repre- 
sent an increasing level of kxxTwledge and a corresponding 
increasing level of intellectual functioning. 

Figure 3 provides an example of such a curriculum 
cycle. It starts with a concrete ex perience such as a 
demonsuation of the concept to be taught. At this entry 
level, it is critical to stimulaic the student's curiosity for 
the topic, it's importaucc, it's validity, and it's intrinsic 
beauty. This can be done through such activities as a 
physical demonstration, graphical modeling on the 
computer, and/or videotapes of its application in the real 
world. If the student isn't iBterested in the subject, she 
will learn it to tbe level nec6cd to pass the examination, 
but her psychic energy will be direaed elsewhere. 

The second step, refiegtive ob servation, can be 
accomplished by getting tbe student to learn the vocabu- 
lary and become aware of the basic concepts, formulas, 
etc. which undergird the concept. This is where the text 
book comes in. Most students only read the text book to 
copy out the homework probldtns or look at sample prob- 
lems for hints on the solution procedure. But if the 
student docs not know the vocabulary for the topic under 
discussion, much of the value of the lecture will be lost 
because it won't be understood. A list of vocabulary words 
or questions to be answered helps direct the students' 
reading and heightens the Vefiection* on what is found 
This activity is also helps buHd the skills of reading and 
information retrieval critical for life-long learning. 

The third step, ah^trad co nfypnt^ltzation- is where 
most instructors start, with derivation of the mathematical 
formulas which describe the physical phenomena under 
discussion. While most instructors do this through a 
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Figure 3 • The Curriculum Learning Cycle 



formal lecture, Karl Smith uses an aJlemaiive delivery 
mo'iel based on cooperative leamixig.(7] In either case, 
the goal is to model for the studem ihe method by which 
professionals use theory to describe the physical world 

The fourth step, active exTx^r imentation. could entail 
two forms of learning, practice problems and laboratory 
experiments. Each has its role to play. The practice 
problems require the student to review class notes and the 
text to tmderstand the concepts and use mathematical 
analysis to apply them in a variety of situations. The 
laboratory provides an alternative, DX>re concxeic, method 
of understanding the conccpu and provides an opportunity 
to explore the approximate nature of the mathematical 
models used in engineering analysis. It also develops skill 
in using technical equipment, maWng physical 
measurements, and performing data analysis. 

The learning cycle starts over again with application 
of the concept being learned to a design problem and/or 
project in which it is tied to other conccpu; a new 
concrete experience, but at a much higher level than the 
original demonsu-ation. The project should require the 
student to deal with the concept urKicr study at the 
synthesis/evaluation level, h can also provide practice in 
the skills of project management, linoe management, oral 
and wrinen prescnuiion, and open-ended problem solving. 

The Assessment Cycle - In parallel with these activities is 
the assessment that can be used at each stage of the cycle 
and the objectives which are being nacasured by f?rb, 
(Figure 4) Some of these assessmcm activities will be 
primarily formative, providing feedback to the student and 
the instnicior on what is known and the rate of progress. 
Others will be primarily for the purpoesc of measuring 
student competence and assigning a grade. It is impUdt 
that each assessment activity also serves as a potential 
source of feedback to the insu^rtor oo the effeciivcncss of 
the learning activity. 



The asscisment activity in suppon of step two, the 
reading phase, could be t check list of definitions and 
equations which the student should locate in the reading. 
If additional structure is needed in order to insure that the 
student does the reading, the instrurtor could give a short 
quiz at the beginning of the class period This might be 
done in order to meet the objective of building skill in 
reading technical literature. 

During a Iccaire, step three, a possible form of 
feedback is a ^i^y^^'on QUffMlftn or problem handed out 
to the students to be worked in pair^L Osiermaim 
proposes that this be done after about 20 minutes, at the 
point where studem attention tends to drifl.(8) Five to 10 
minutes spent on this acuviiy is rewarded by increased 
attention, better ujodersunding of the material just 
covered, and feedback to both the student and instrurtor 
on the level of ui>cicrstanding of the topic The discussion 
question also gives a means of taking role in a large class 
and stimulates cooperative learning between students. 



Pesr assessment 

Log book ^^^v 
Project Report^ 
wrintn.»f^ Rnalsxam 
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Figure A - Tbe Assessment Cycle 

Use of honicwork for assessing learning, step four, 
is common in engineering courses. To improve the level* 
of feedback to the students, one might have the students 
grade one problem on each other's homework, using a 
grading scheme provided by the instrurtor. This reinforces 
the importance of doing the problems, and provides 
smdents with a model for effective solutions and examples 
of a variety of approaches. This technique was used at the 
University of Strathdyde. with a significant improvement 
in student test performancc.(9] 

Peer marking can also be used on midtenn cxamin- 
auons for the same reasons. While additional time is 
required ir preparing the examination and the solution 
marking procedures, instrurtor time is saved on marking 
the exammations and smdents benefit from the more 
effective fcedback,[10] 
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The Uboraiory report is a logical means of obtain- 
ing feedback on student learning. However, the convcD- 
♦ional rcpon provides little insight on student knowledge. 
K Most students can complete the laboratory assignment and 
) ncet the counc requirements, given enough time. If the 
students arc ^-orking in pairs then it is possible that one 
student is canying the other. This is an area where the 
skill objective should be clear and distinct from ibc 
knowledge objectives and should be measured separately. 
Skill in using equipment, taking accurate measurements, 
precision in calculations, and effective writing are all 
required for cffeciive. laboratory performance. Under- 
standing of ibc physical concepu is equally imponani but 
quite scpamc. An appreciation of safety regulations. 
responsibilitN' for ones own cxp^ nmentaJ results, and the 
importance of experimental work for creative engineering 
practice are aiiitudinal objectives which require separate 
attention. 

Two other aaivities besides the lab report which 
can help in measuring these skills and attitudes are the log 
book and the debug test. A log book provides an oppor- 
tunity for ibc student to keep track of his progress in 
learning the topic under study, particularly in the 
laboratory. But it should be a record of all learning that is 
going on. Optimally, the instruaor will read the log book 
and discuss m contents with each student Praaically, 
there is little time for this. However, someone besides the 
student sboukJ read it and should make comments on the 
entries, both in order to improve the use of the log book 
and to help the student appreciate the progress that is 
being made. Donald Woods makes use of the logbook to 
help students develop self assessment skjlls.[ll] 

The debug test, or lab practical, is an attempt to 
determine vk-bether the student really understands how to 
use the equipment. It can involve nothing more that 
setting all the dials on an oscilloscope to zero and asking 
the student to set it up to accomplish a particular sweep 
and gain. Or, at a more sophisticated level, the instruaor 
can introduce a bug into a piece of equipment and ask 
the student to determine what is wrong. This activity will 
separate the students who understand the equipment from 
those who rely on others for help in the lab. If students 
understand that this is an objective of the lab and that they 
will be tested on it, they will take a different approach to 
using the lab equipment 

The design problem/projea step provides an oppor- 
nmity for a much wider range of assessment Too often in 
projea courses, the objectives and methods of assessment 
are poorly defined, so the student is working in the dark. 
Frequently, the instruaor is also unsure of bow to assess 
design/ project work. Approaching the assignment from a 
professional perspective can help. If this were an assign- 
ment to a junior engineer in industry, what would be 
measured? How would the assignment be nude? What 
would be considered an accepuble outcome? How would 



the engineer be expcaed to report the reseats of his work? 
To whom? The assessment should be fraised such that it 
measures performance at the synthesis/csuiuation level 
Many design projects only require analysis. Problem 
defmiiion should be a major pan of the assignment 

As in the laboratory assignment, the assessment can 
measure skills in a variety of areas, such as: group 
participation and leadership, projea management, time 
management, oral presentation, report wriiing, and 
professional attitudes. Some of these may be measured 
purely for feedback and to support skill development, 
particularly the first time they are mcasiuTsd Without 
some form of assessment, they may not be taken seriously. 
Funhenmore, many students will not see tbem as skills 
which can be developed with praaice. 

If the other forms of assessment lisicd above are 
used during the course, the final cxamiiurion can be 
purely for the purpose of accreditation. The formative 
assessment included in the other aaivities uiU provide 
adequate feedback to the student 

The curriculum and assessment actK-ities selected 
above are purely illustrative. The types of assessment 
aaivities, as well as their scope, will depend upon the 
learning objectives set for the course. Careful analysis of 
the results of each assessment activity can provide the 
instructor witt valuable information on bow to improve the 
course and which forms of assessment arc most useful to 
student and instruaor. 

FEEDBACK FOR COURSE DEVELOPMENT 

While it is implicit in the description of the 
curriculum/assessment cycle given above that the 
instruaor is obtaining feedback on the effectiveness of 
learning aaivities, there are two specific means of 
obtaining feedback that should be raentioDcd. I have 
found the use of class represcnutives and midterm course 
evaluation sheets to be simple and effective ways of 
obtaining feedback during the semester ^iiich permits me 
to adjust the course to meet the needs of a particular class. 

Gftss Representatives - Most students are reluctant 
to inform the instruaor if there is a pioWem with the way 
a course is organized or is being uught There is always 
the assumption that if a student complains, that part of the 
fault is with the student, not the course. To circumvent 
this problem, I ask for three student vohuneers to serve as 
class represcnutives at the first or second dass period. 
Qass eleaion of representatives proved burdensome. 

I meet with these students about once a month, 
starting about two weeks after the be ginning of the 
semester. Because -hey are representing the class, they do 
not have to assume personal responsibiliiy for their 
comments. It is important to report back to the class on 
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their comments after the first meeting. iJii any changes in 
Krhedule, etc, should be attributed to the student input. 
After the first meeting, other students in the class will 
begin to seek out a representative If they have problems. 
By Ibc end of term, if there is a problem, the represent 
tatives will come to see me oo their own initiative. 

The class representatives frequently become the 
instructor's advocate with the rest of the class. Because 
they have taken the time to get to know the instructor 
personally, and have learned about the rationale behind 
the course, they can answer many student concerns before 
they become a problem. 

Midterm Course Evaloalioo • A single sheet qucs- 
tioimaire with room for short answers in response to 
quesUons on course organization, presenuiion, etc^ 
permits the students to identify what they like and don*t 
like about a course. A Likert Scale ( 1 to 5) does not 
provide useful feedback in this case. The midterm 
evaluation identifies ways that the course can be modified 
to improve student learning in ^at particular claits . It is 
important to give students time in class to fill out this 
form. If they are rushed, or if they complete it out of 
class, the answers will not be as informative. 

CLOSURE 

How much assessment b enough? Look at your 
research and ask yourself the same question concerning 
the level and magnitude of results required to support your 
publications. We should be aMc to defend what we are 
doing in the class to the same level that we can defend our 
rcscarcL Otherwise, we are Dot meeting our professional 
responsibilities. 

RICHARD S. CULVER 
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ducating for Maturity: Perry's Model for 
Intellectual Development* 



CULVER 

Thorria^l Watson School of Engineering, State University of New York, Hinglumuon, NY 1 390 L U.S.A. 

' Modern engineering education programs are very cffcaiw at icadiinii smdaits the nu lunnii 

material of the engineering profession. They ore iess tffcaivc at prvparinv^ tlicntfor tin- inanii c t *j 
engineering, where the problems are typically complex and ilNlcftnvd, and liaxv niilml. 
nontechnical components. Effective engineering practice calls for a prnfcsaiomtl wlut i\ 
intellectually and ethically mature, who can deal with complex issues, make cjji'cmr nw 
authority, and assume responsibility for his/her decisions. l*err\''s model for inu'llcnmil 
development provides an effective description of intellectual maturity ami .ut^ncsix the types of 
instructional activities which will stimulate intellectual growth. In diis papa, the auilnu- 
compares the challenges faced by the student in a traditional engineering education pni}itain with 
■ those faced by the professional and shows that, since the challenges are different, the students cir 
not being encouraged to develop the critical skills and attitudes nupured by the i»'oles\iotud. \ Viih 
Ferry's model us a theoretical basis, he then descril>es educational artivirie^ wliirh con Mitnidttic 
such development and lists some engineering programs which lum incorpontwd these arit\ itie\ 
into their curricula. 



MATIURITY; "The stale of being fully devel- 



body and mind"-— Webster's Dictionary, 



INTRODUCTION 



^lopcdiin 

if; 

'WHEN asked what attributes he would seek in a 
hew ^r£\duate, a practicing engineer started the list 

or she, must be able to mature rapidly. We 
lew graduates responsibility much more 
rapidly than in the past " 

iAs i|e>' 
profess 



plored the characteristics of the effective 
onal, this attribute seemed to encapsulate 
; the b'th^rs. 

i ^ Atfcebting this as a goal presents a significant 
^ challenge to engineering schools, as it suggests that 
i wc should build an emphasis into our programs 
\ that cuijrently seldom exists. Since there is far more 
I to teacfi than can possibly be covered, we must 
' carefully zsstss what is the optimum use of our 

4-yekr window of opportunity when preparing our 
; imden^ for professional practice. 
\ Inj p!articular, it challenges the professor lo 
! expand his role in the education of the student. On 
' most c|>unts, we do a good job of mentoring the 
; full-time graduate students, serving as professional 
;i rolejnibdels, and engaging them in the learning 
''I procesL But only a small percentage of our 
•i students go on to graduate school and most who do 

are preparing themselves for that narrow portion of 
} the engineering profession dealing with research. 

For most undergraduate students, contact with the 



•Paper received 27 I>icember 1986. Final version accepled 
27Jinmry 1987. 



professor is superficial, constrained by ihe boun- 
daries of the class period and formal. 

What are the characierisiics of the mannc pro- 
fessional engineer? Innumerable studies havo 
addressed this question. For the purposes ol iliis 
paper, 1 will use the following: The effeciivc prt)- 
fessional: 

(1) has mastered a body of knowledge i\ni\ ihc 
corresponding skills central lo his iieitl (>! 
practice, 

(2) exercises good judgement in making deci- 
sions, 

(3) uses authority as source of experiisc, 

(4) assumes responsibility for decisions, 

(5) is capable of thinking critically ami com- 
plexly, 

(6) deals with problems in their total compiexiiy, 

(7) uses reflective thinking and sell-evaluaiiiMi, 

(8) has an effective and socially responsible 
value system. 

In eariier papers. Culver [1, 2| has contended 
that a person with these characteristics is cap,ih!e of 
a high level of intellectual functioning, as dellned by 
Perry |3|. In this paper, I will compare !he 
similarities and differences between the crieeiixe 
student and effective professional with Peiry's 
model, propose some goals for t!»e cdueaiio:i;t! 
program, and look at the role of the prorcss(n in 
achieving them. 

PERRYVS MODEL 

In the 1950s, William Perry, Director ol i!)e 
Bureau of Study Coimsel at Marvard Coileiic. 
undertook a longitudinal study to deiernnnc "whni 
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happens (o n studenl exposed to four years of 
lihcral an.s education?" The research occurred 
within ii.c framework of an educational philosophy 
which claims thai education should nourish the 
process by which a student approaches complex 
i.sMics. This increasing sophistication occurs in 
pari as the individual matures*, but also in response 
U) stimuli in his environment. Thus, college stu- 
dents will mature intellectually when exposed to an 
environment whicli fosters critical thinking. 

J't*nyssludy, based on thccognitivc-development 
views of Piaget |4|, started from an empirical base, 
lie began by focusing on how college students 
respond to ilio intellectual and mora! atmosphere at 
liarvard College, an atmosphere that appears to 
present constant ciialienges to the students' un- 
examined assumptions about the world around 
ihcm. After 4 years and 98 recorded interviews, 
Perry and his colleagues observed that there 
appealed to be "... a common sequence of 
ehallengcs to which each student addressed himself 
in ins own particular way." This led Perry to 
h)rnuilate a model for the sequence of intellectual 
developmen I. l ie then expanded the study to test his 
hypothetical model under better controls. 

The following is a paraphrase of the stages, or 
•positions; of intellectual development identified by 
i^crry: 

Dualism 

rosiuoii i . Students see the world in polar terms 
oi we-right-good vs oiher-wrong-bad. Absolute 
right answers exist for everything; the answers are 
known by Authorities whose role is to mediate 
(leach) ihcni to students. Problems are solved by 
following the word of an Authority, rules, tradition 
or ihc n-nn. Knowledge and goodness are per- 
ceivci. as quantitative accumulations of right 
.-Hiswers thai are collected by hard work and 
<ilK-dicncc. 

Positkm 2.Students begin to perceive alternative 
views as well as uncertainty among Authorities, but 
Jiccouni Iwr ihcm as unwarranted confusion among 
paorly qujililicd Authorities. They may decide that 
uaeeruiinty is simply a game devised by Authority 
•m) wc can learn to llnd ihe Answer for ourselves ' 
1 heir arguments are often illogical, and they do not 
uiulei siand ihc use of evidence. 

Miiiti/tlicify 

I'osiiian .'^.Siudejitsscediversity and uncertainty 
as Icgiiimaic. but still temporary, in areas where 
Auihoniy'.Masiri found the Answer yet/They begin 
(o sec Author iiy as ^biased' or arbitrary in that they 

arc graded for 'good expression.'The apparent lack 
o( standards is pu/.xJing.and many students begin to 
view all arguments as equally correct or biased, or 
bo;h. wiih clccisions based predominantly on 
pcrsoni'.l belief. The hard sciences and mathematics 
seem better understood by Authority than humani- 
ties or social sciences. 



Position 4. (a) Students perceive legitimate 
uncertainty (and therefore diversity of opinloil) to 
be pervasive and raise it toa realm of its own irj which 
'anyone has a right to his own opi nion,* as opposed to 
Authority's realm, where right and wror g Istill 
prevail. Or (b) students discover qualitative, iklitiv- 
istic reasoning as a special caseof 'what They want' 
within Authority's realm.ln either case, student^are 
suspicious of the *iruth' of any evidence lorj an 
authority's opinion, and often deny that opinions 
can be objectively evaluated. 



Relativism 

i Position 5. Students 'perceive all knowledge and 
value (incJuding authority's^ as contextual and 
relativistici They subordinate rights-wrong Iprpb- 
lems to the status of special coses, in a frame ofirefer- 
ence.They begin to recognize ihat all points of view 
are not equally correct and may argue on the basis of 
evidence that one is more likely. \ .. 

Position 6. Students recognize the need to farient 
themselves in a relativistic worid through peifsohal 
Commitment (as distinct from unquestionfed or 
unevaluated commitment). While they can rily on 
evidence, they still may not synthesize that eviierice 
for.themselves but rely on the synthesis of otljersi ; 

Commitment in relativism 

i Position 7. Students make an initial Commit ment 
in some area, such as career selection, vilues, 
religious belief etc. However, they examin s the 
views they endorse based on logical evaluation Tof 
evidence, the opinions of experts, as well as 
reasoned conjecture about 'what appears to be true.' 

Position 8. Students experience the implications 
of Commitment, and explore the subjective and 
stylistic issues of responsibility. 

Position 9.Students experience an affirmatiDn of 
identity among multiple responsibilities and r^ize 
Commitment is an ongoing, unfolding activityl and 
that their own views may be incorrect and may leed 
. to be reformulated in the future in light of additional 
evidence. 

According to Perry, one of the major aciom- 
plishments of college students is to progress (rom 
a simple, dualistic view of life and knowledge to* a 
more complex, mature view which is also relktiv- 
istic. Until this transfer occurs, students arel not 
'their own people' in the sense that their va ues, 
attitudes and life goals are borrowed from the 
influential Authority figures in their eariier devel- 
opment. Note that when the transition occurs the 
'Authority' in the upper case (unchallenged) 
becomes the 'authority' in the lower case, whc'the 
student has evaluated and accepted as having tiic 
best answer at the present time. I • 

While Perry found that most Harvard students 
had reached position 6 by the time of graduataoii; I 



subsequent studies have indicated that, in n,c 
academic environment prevalent on most moc cm 
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. i campuses; there is relatively liiile intellectual devel- 
[ opment i n the college years jSJ, Most students enter 
lithe ^university in position 2.'Or 3, and leave in 

• position 4,? which probably is not what most 
. educaioi s'would prefer. This position is character- 
• izedby: ' 

• I r.'.i 



(4). 



a tendency to treat all opinions as equally 
gcod; . - , j 

little ievaluation. of alternative views of an. 
isjuej: . !; I 

a 1 endency to hold one|s opinions largely on 
th ^ basis of whim and unsubstantiated bel ief , 
a lesitancy to take a stand or commitment' 
based on evidence and reason [6]. ! 
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Interestingly, position 5, where the transition^ 
from miltiplicity to relativism occurs, was rarely 
identified by the students while they were in that 
position. Only in retrospect could they identify this' 
fundamenia) change in their way of thinking. 

Apparently, the transition process from concrete' 
to abstrict i thinking is perilous, requiring con-! 
cemed c/iallenge and reinforcement by the instruc- 
tor. Newlell recognized that during this transition,' 
thestudentsicommitmentis noticeably reduced. At 
the lowe- stages of Perry*s model, commitment is 
based !upon Jfaith' in Authority and the rightness of 
Authorit/*s positions on key issues, As the student! 
grows* that faith is slowly eroded by unreconcilable 
conflicts Only when the student gradually rebuilds; 
commitiBent through a process of personal analysis' 
will he )e able to convert what was faith into' 
personal and self-committed belief (7|. 



This p 
where! a 



ocess is illustrated schematically in Fig. li 
the lower stages, everything is black or! 
white an J a ''dominant Authority figure has all the 
right answers. As diversity increases, commitment' 
declines intil the transition at position 5 creates a* 
new, .more open viewpoint. A need for commit-; 
ment is realized at position 6, initially attempted at" 
position 7, evaluated at position 8, and established 



INTELLECTUAL DEVELOPMENT 
Fig. 1 . Perry's model nf inlelieci iial tlevolopmeiu. 



as a process of coniinual revision and development 
at position 9. 



EFFECTIVE PERFORMANCE IN AN 
ENGINEERING ENVIRONMENT 

Perry's model provides a tool for looking at how 
we prepare engineering students for pro1essi(^niii 
practice. Let us approach this task by comparing 
the challenges and performance of the succcssrul 
student with the successful practicing engineer, on 
the premise that professional preparation is Ik\si 
accomplished by having the student model profes- 
sional behavior. Table 1 presents a comparison of 
.the measures of success for the student and ihc 
professional. It shows that tlie student is expected 
to learn prescribed solutions to classes of prob- 
lems. He is measured by the professor in his ahiliiy 
to perform these procedures quickly and correctly. 
By contrast, the major challenge for the profes- 
sional is determining what problem needs lo he 
solved. Speed is not as important as insuring that 
the problem is properly defined before solution 
proceeds. The problems faced by the professional 



Table 1. Criteriu for effeciive performance 







Engineering student 


Professional engineer 


(1) 


View of knowledge 


Discrete facts, 
procedures, subjects 


Connected a>ticepts 
and facts 


(2) 


Critical 
kr.owledge 


Algorithms and facts 


Heuristics lor decision 
making 


(3) 


View of insirucior/ 
supervisor 


Source of kiu^wlcdgc, 
authority 


Source of expcn isc, suppoi i 


(4) 


Source of 
Evuluuiion 


Instructor 


Self and peers 


(5) 


Approach to 
learning 


Responsive 


Proactive 


(6) 


Work/learning 
environment 


Competitive 


Coopcriiiivc 


C) 


Critical problem- 
solving skills 


Efficient at solving 
structured problems; 


Flffeciiveai deriiiing pioblciiis 


(8) 


Goal of work 


Good grades 


Usable product or service 
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arc typically complex and muliifaceted» requiring 
cooperative input by other professionals and self- 
education in new areas as the solution progresses. 
While the student defers lo the professor as the 
source of knowledge, right answers, and evalu- 
ation, the mature p/ofcssional selects authorities 
on the basis of expertise, but assumes responsibiljty 
for evaluation of his own work. 

While exccjitional work has the hallmark of 
creative genius, whether produced by the student 
or the professional, it can be seen that, in virtually 
all the areas listed in Table 1, the approach' and 
criteria are different. By comparing the lists in 
Table I with Perry's model, it is apparent that the 
challenges jircscnted toAhe student reinforce the 
funeiioning in lower, dualistic positions] while the 
successful professional jis called on to fiinction.in 
the higher, relalivistio . mode. If the i academic 
challenge does not encourage performance at the 
higher levels, the sttident has no incentive, for 
intcllcciuai growth. • 

In his book, Vic Rcjlcctive Practitioner, Schori 
|S| describes how the effective professional*, uses 
rcfloctivc thinking during the critical phases iof 
problem formulation. The reflection lends. to be 
intuitive, nonlinear aiid, at times,* nonrational. 
Through practice, he; develops, the., judgement, 
needed to expand the variety . of i problems 
addressed and the effectiveness of the solution. By 
ctMitrast, the applied scientist uses a more elegant, 
rational approach to the development of analytical 
solutions. Schon distinguishes between{-_ the 
engineering professor, who he sees as an applied 
scientist, and the practicing engineer in terms of 
convergent and divergent thinking. Since it is the 
applied scientist that is recognizedj by the i edu- 
cational establishment, it is his approach ' to 
prt>blem solving that is typically presented to the 
student in the classroom. While competence in 
analytical solutions is also a critical skill for the 
practicing engineer, it is but one of many skills 
needed for effective professional practice. 1 con- 
tend that our students also need practice in dealing 
with open-ended, ill-defined problems in order to 
start tlcveloping the art of the practitioner and the 
ability to h nidle complex systems. 



KKST RUCrURlNG THE CURRICULUM 

if we wish to improve our students* preparation 
for |)rolcssional practice, we need to identify those 
skills and attitudes which are underdeveloped in 
our current graduates. The report of the Future 
Ciradiiale Study at the Colorado School of Mines 
|9| acknowledged that the graduates -were, tech- 
nically competent and hardworking.. However, 
along with graduates froni other engineering 
schools, most graduates . , lack good communi- 
cation skills, even in writing. They are unfamiliar 
with working in interdisciplinary groups . . . cannot 
investigate an unfamiliar problem starting from 
ftinilanicntals, and mipart only limited knowledge 



tie 



relucta le 



oU and interest in, their responsibilities to 
social, political and natural surroundings J . 
(heavy) curriculum and insularity of departhi€}nts 
leads to the 'credit syndrome', in which each ^oi^rse 
is seen as a hurdle to be jumped and left beh^nd^ 
The narrow nature of their assignments J 
cook-book approach j to the solution': of 
defined problems, at which the graduate is 
but also to a lack of curiosity and a 

. approach broader, unfamiliar, problenas 
fundamental standpoint.'' |i 
I **1rhe reasons for these limitations are not 
find. The growth of public education; with 
resources and increasing enrollments, has 

• oped an approach to education which empli; 
memorizing answers and solutions! to J 

. :problems.'Multiple-choice tests, lectiiresi. 
istudents,! and depersonalization of.th^' 
environment have led the students to see 
as a means to an end rather than an end to be 
in itself. Emphasis has'been placed on fitting 
subjects into less timeiand more details.in 
course. We, hear more about the studeii|t 
finished the (prescribed program in 3i 
the student who chose:to reduce the course 
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years 



lorder io, learn the material more thoroughly; 
. jexplore areas. of interest outside those reqtii|e4 
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can 
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tear ning 



.. i the degree.)!; r • ; iji = -i > • •» j 
' I iiWhat. can we do? Some features, of 
isionally-briented program are:- = 

■ i I ■ '■ ' ' i ' 

i 1 (1) give the 'student more ownership 
' " ■ ' learning,'! ; i- : " : 

(2) reduce the prescribed work 
mcrcase project emphasis, • [ 

(3) create ■ *mar/ccr events' which 
meaning to the learning, ' ■ 

(4) have a mentor, for each student,-' 
; (5) structure the educational activities tc 

: late increasingly 'mature' performai 

(6) train the student in [performance 
^ atlon, • :i '1 

(7) involve students in each other's I 
I (8) provide projects for students with 

! j sional context, ' " 'j ' 

; i (9) create activities thai involve risk 
: j and assuming! res/7on5i^i/i/y for 
i j sequences, i| " ' j 

1(10) \x%^' developmental instruction'* ion 

' design educational activities. *! 

. . ,1 * . ' 1 

Ow/icrj/ii/?— Ownership of learning imp! 

; the student sees himself in control of. 
j cational program and, because of a positive 
j towards it, becomes actively engaged in 
ing process. The other features hsted jbel 
suggest how this can be accomplished., j ,v 
"1 Reduce course workload:— A common 
thumb is that a. student can be expected 
' 55 h per week. Assignments need to be 
: throughout the program so that they can 
! pleted within this time frame. If the worklo)ad 
heavy that the student cannot master one 
befot w passing onto the next, it leads to 
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Icanthtii wliicli is unsatisfying and results in a 
sinrival mode of operation in which Iciiming runs a 
pDor second to passing the course. 

f'mjccf em/)hasis-To combat the ^credit syn- 
drome, students need to know how they will use 
the subjects being studied in math, science and 
humanities. This can besl be illustrated by having 
ihcm involved in-*doing engineering' at an early 
stage, even if the projects are technically trivial. In' 
elass discussions during the project, the instructor 
can explain the relevance of the other courses to the 
educational background of the professional, but m 
the context of the project.! , ■ ' .■ 

Marker c'r(f//Av— Students arc more likely to 
leiain a new procedure or concept if it is tied to an 
cveni wliich has personal signincance to them, thaf ' 
stands out in their memories. This event frequently 
is the basis for new insight, a rearranging of think- 
ing, and major jumps in growth. It can be positive or 
neuaiive. (F'ailing a course is a marker event for 
most students.) Since the/marker'.is the student's 
response to a stimulus, not the stimulus itself, we 
cannot force our students to experience markers in 
oiu courses, but we can crcjite an environment in 
which they can happen. The more intense the 
response, the longer the lesson will be i remem- 
bered. Industrial tours, student design competi- 
tions, an impressive laboratory demonstration^ or : 
vhc completion of a major project are likely sources 
of marker events. A course designed to include two ■ 
or three activities with marker potential is more" 
likely to influence a student's career than one 
without. j ; . . 

A/cv//r;r.v— People recognized as ! successful 
uniformly identify the mentors who were critical in 
shaping their careers. The typical advising system 
used in engineering schools is not designed to 
develop a mentor relationship between student and - 
fncuUy. Given prevalent student-teacher' ratios; it 
is unrealistic to put this burden entirely on the 
shouUlers of the faculty. But upperclassmen, 
properly trained, could start the process with fresh- 
men. Professionals from industry could assist on a 
Uii: Brother basis. The student needs^ help in 
n^.akinu meaning out of a collection of courses. 

rrojc.ssianni /^/-o/cr/.v— Left to pick their own 
j^rojecis, students tend to select something that is 
narrowly defmcd, using previously developed 
skills, or a prc^ject so broad that it can't be managed. 
Carefully selected projects, submitted by an exter- 
••ai clien't. can provide real-life experience. These 
should have a social or economic context which the 
Mudents are required to address as part of the 
solution. Project teams with students from different 
acadeniic years can provide internal mentoring. ; 
Ihc clicni-based project has strong marker 
nolcnlial. 

hinrasinii matmvy—h is unreasonable to 
expect a freshman to function as a self-managed 
learner. (For discussion of self-managed learning, 
see Culver |10, 11].) But since effective engineers 
MUST do so, the transition would ideally occur 
during the 4 years on campus. The process can 



start- with a fre,shman project or design course). By 
the senior year, most courses should be structyred 
to call on 'mature' performance, with the inslruetor 
as coach instead of manager and judge. | 

Performance evaluation-'Ont of the marMs of 
the mature professional is the ability to evaluatfe his 
own; work and assume responsibility for it. ^eer- 
evaluation is important in developing the stud ent's 
understanding of professional relationships. Stii- 
dents should start with quantitative evaluation and 
progress to qualitative evaluation of reports, oral 
presentations etc. 

j Cooperative learning'^Tht academic environ- 
ment is primarily competitive, but the critical 
phases of professional performance nornally 
occur in design groups, which must work cooj era- 
tively. Competition between student project t< ains 
can give the students the opportunity to experi Jnce 
both concurrently. t 
: Risk and responsibiliry—PSsk taking and as! lurn- 
' ing responsibility for results are critical qualities of 
the professional. In most cases, the risk in!an edu- 
' cational activity must be dissociated from the grade 
; received or the students won't *play the gime.' 
■ Academic .risks do not accurately model the risks 
faced by the professional. Projects, simulation 
games, and student competitions are good ways of 
; exposing students to risk.taking. j I ^ 

; I Developmental ww/rwcrio/?— -Based on- Parry's 
=^ intellectual development = scheme. Cornfield and 
' Knefelkamp's developmental instruction n odel 
provides a procedure for selecting and designing 
educational activities to achieve an optimum 
balance between support and c/ifl//e/;ge,i Experi- 
ence has shown that this approach to curricilum 
design can significantly increase the rate of sti ident 
' development. Fitch and Culver presented a v^oiik 
. shop on Developmental Instruction at the 1?°' 
Frontiers in Education Conference, which 
reported in the Proceedings j 1 2J. f 



1985 
is 



THE ROLE OF THE INSTRUCTOR 



tipat 



BEST COPY AVAILABLE 



^ From the statements above, it is apparen ^ 
thei professionally-oriented program enviiaged 
would pose a different role for the prof(isspr. 
Instead of functioning as an 'independent sub- 
contractor' who lectures on a given topic, v^ith a 
student assistant to do the grading, the prolesior 
would be a member of a management team \^h\ch 
supervises the student's learning. Some oi the 
activities suggested could be built into an indi\ idual 
■ course, but since student . growth to rm iture' 
prqfessional functioning is a gradual process oyer 
the; 4-year period, it will occur most effectively] as 
the result of planned growth, rather than a sei ies of 
disjointed efforts by individual professors. Tliis 
calls for a redesign of the curriculum with tht gbal 
of producing a graduate prepared for profesiional 
practice, as well as for research-based graduate 
studies. It would be a program designed to ernpha- 
sizc general professional skill development, rather 
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than llic specific technical skills needed during the 
firsl year on the job, allhough some lechnical skills 
would he covered as practice in self-education. 

In his role as a learning manager, the professor 
would work with the student through a learning 
contract, serving as coach and mentor. While a 
course in such a program might look similar to 
conventional courses from the outside, the" inter- 
action between student and professor and their 
expectations for the course would be different. If 
llic sUidcnl is to gain control over the learning, the 
in.siruclor would have to give up some control. This 
would require more lolerance on the part of the 
insiriiclor, since the student will make some 
mistakes as he explores the scope of his responsi- 
bility. Il will also demand more of the professor's 
energy, since il lakes more effort to manage a 
sy.sicm that you do not completely control. 

As control is gradually shifted to the students, 
evaluation should begin to look and feel more like 
that which occurs in a professional environment. 
The professor would be responsible for monitoring 
the quanlily of work being assigned, as well as the 
an)Ounl and quality being produced. In the profes- 
sional work place, the manager is responsible for 
insuring that assignments can be completed 
according to schedule, .since he is evaluated on his 
ability lo manage the projects in his charge. This 
type of accountability is alien to most higher 
education, but could be the basis for insuring 
responsible instruction, as well as effective learn- 
ing. 



SOME EXAMPLES OF PROFESSIONALLY- 
ORIENTED PROGRAMS 

This may sound like an overwhelming job, but it 
has been achieved to varying degrees in some 
innovative engineering programs in the United 
Slates. 

WorcliesterPolylechnic hisiitute (WPI) restruc- 
tured its curriculum in 1971 ;i3).TheWPI PLAN, 
as it is known, emphasizes competencies, indi- 
vidual freedom within a student's curriculum, self- 
iniiialcd invesligation. and new in.slruclional 
methods which join students and faculty in a learn- 
ing partnership. The four major activities of the 
PLAN include: 

( 1 ) a competency evaluation of a major field of 

sindy, 

(2) a major qualifying project (MQP) which 
inlcgraies formal academic studies in the student^s 
mwjor neld through an in-depth research project, 

(3) an interactive qualifying project (IQP) which 
focuses on the interactions of technology with 
socicly and human values, 

(4.) a sufficiency in a minor area. 

The MQP and IQP provide the practical experi- 
ence required for professional preparation, while 
the procedure for selecting courses and minor give 



student ownership of his learning in ii manner 
uncommon in undergraduate education. 

Harvey Mudd College established the Engineer 
ing Clinic in 1 960, where junior and senijDr under- 
graduates work with a masters candicate on a 
client-submitted project, on which the graduate 
student serves as a project leader |14). As, in the 
WPI program, the projects are carefully screened 
to provide appropriate learning opportunities. 
Since the institution guarantees complet on of the 
project, students may leave the institution ibefore 
the project is complete. Permitting students to 
work into increasingly responsible positi<ms within 
the project team models professional development 
of the graduate engineer. { l 

A similar Design Clinic was develops d by Lee 
Marrisberger at the University of Aiabama.!Senior 
mechanical engineering students do individual 
' projects and participate in exercises selected to 
teach professional skills during the fall semester 
and then participate as members '^f project teams 
to solve client-based problems in the; spring 
semester 1 15]. \ 
' The guided design principles deve oped by 
. Wales and Stager are uniquely successfu in teach- 
ing a process for problem solving anc^ decision 
making (16). Guided design exercises arp particu- 
: lady valuable as an introduction lo. onen^ended 
1 problem solving. ■ 1 i ' 

The EPICS Program at Colorado School of 
Mines encompasses developmental instruction 
concepts into the first 2 years(17). Afoursemester, 
10 credit hour course sequence, EPICS integrates 
instruction in graphics, computer programming, 
technical writing, and open-ended problem solving 
. in a manner which provides a context f^r jupper*- 
division engineering courses. The fact 
instruction is spread over four semesters 
tant because it paces the student's persoi al jmatur 
ation. 

In recent years, several other ertgineenng 
schools in the United Stales have begui^ 
some" of the ideas presented in the 
described above. 
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Vicious cycle of preferred learnings 



|. S3 



ylM. 



371 



ll 



' . The edlicational 
ambitious 



concepts proposed above are 
perhaps even threatening to some» 
becaus^ they call for planned growth and change of 
both the student AND the professor. Students 
prefer to learn in a mode that is comfortable; the 
traditio'na lecture format is comfortable because it 
is familiar and responsibility for the learning rests 
primarily vith the instructor. The professor prefers 
the traditional mode because he learned well that 
way. O^hejrwise he would not have made the grades 
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MANAGING CHANGE 



to go to graduate school and on to a position on the 
faculty. This is the viciotis cycle of preferred learn- . 
ing styles (Fig. 2). j 

However, if the educational program is going lo 
improve, it must change. This will require change in 
the way we teach, and the change will involve risk. It 
is important, therefore, that the ndmini.strnilDn of 
the educational institution also assumes its i espon- 
sibility for improvement of the program through 
innovative development. This can best he done hy 
openly supporting faculty who arc willing lo risk 
learning new ways to leach. 
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INTRODUCTION 

"IT IS strange that we expect siudents lo learn, 
yet seldom teach them anything about learning. 
We expecl students to solve problems, yet 
seldom leach them anything about problem 
solving, and, similarly, we sometimes require 
students to remember a considerable body of 
material, yet seldom teach them the art of 
memory. It is ame we made up for this lack . . ." 

II, p. 9?;. 

Educationalengineeringisbasedon recent develop- 
ments in knowledge engineering and cognitive 
science. The contribution of knowledge engineering 
(and the broader field of cognitive science) for 
improving learning efficiency and effectiveness will 
be discussed along with the implications and impor- 
tance of these ccmsiderations for engineering educa- 
tion. 

Drucker coined the terms efficiency (doing the 
thing right) and effectiveness (doing the right thing) 
in reference to business management |2]. The terms 
appiy equally appropriately to learning efficiency 
(enhancing the rate of learning) and learning 
effectiveness (enhancing the mastery and retention 
of facts, concepts and relationships). Students* 
learning effectiveness and efficiency can be 
enhanced by providing them with strategies that 
promote learning how to learn. 



LEARNING HOW TO LEARN 

The concept of "learning how to learn" was first 
articulated by Bateson |3| who termed it "Deutero- 
learning** and il was associated with the then new 
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science of cybernetics. Also called "double-loop 
learning" this form of learning involves changes in 
the governing variables, in contrast to "single-loop 
learning", which involves learning of new strategies 
to achieve existing governing variables (4|. 

The process of "learning how to learn** is com- 
monly referred to in the cognitive science literature 
as "metalearning". The prefix "meta-*' in this usage 
is supposed to be analogous to its use in the term 
"metaphysics" and means "going beyond", "on a 
higher level", or 'transcendent*. In a like manner, 
"metaknowledge" is used to refer to the structuring 
of knowledge. Metalearning and metaknowledge 
are two different but interconnected concepts that 
characterize human understanding. Learning 
about the nature and structuring of knowledge 
helps students to understand how they learn, and 
helps to show them h(nv humans construct new 
knowledge. Novak and Gowin |5| describe specific 
strategies— concep* mapping and V-hcuristic— for 
helping students learn about the structuring of 
knowledge and the process of knowledge produc- 
tion. Concept maps are intended to represent 
meaningful relationships between concepts in the 
form of propositions. They are (wert, explicit 
representations of the concepts and propositions a 
person holds. The V-lieuristic is an aid to help 
students understand the contribution of and the 
relationship between conceptual and methodo- 
logical views in making sense of events or objects 
ob.served. 

The approach of this paper will be to follow the 
engineering method, definetl by Koen as '*ihc 
use of heuristics to cause the best change in a 
poorly underslootl situation within the available 
resources**. Meurisiics liom cognitive science arc 
reviewed thai assist in making the best change in 
studenls* learning eHeclivcncss ant! elTiciency. 
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ROLE or HEURISTICS 

Hcurislics arc csscmial for a Jiscussioii ol 
Icarninu how to learn since ihere arc no clcar-cul 
algorithms for second-order learning. The word 
"heuriskin*\ meaning ^'serving to discovert Heur- 
istics have become very po))ular in the cogmlive 
science literature, and interest is growing once 
again in the problem-solving literature |7,K| and 
the engineering-method literature |y|. 

One of the most prolific promoters of heuristics 
was George Polya. Polya \\0\ proceeded from 
three assumptions: (I) *Mieuristic'' means **guidmg 
discovery'^ (2) solving a problem certainly involves 
a bit of discovery, hence it involves an heuristic 
process; and (3) investigation into heuristic pro- 
cesses cannot aim at finding infallible rules of how 
to make discoveries. Polya's work in heuristics is 
aimed explicitly at leaching the young how to be 
better problem solvers III]. The main usage of the 
word "heuristic" at present is as an adjective in the 
sense of "guiding discovery" or ^'improving prob- 
lem solving". Although difficult to define, heuristics 
are easy to identify using the characteristics listed 
by Koen 16): heuristics do not guarantee a solution; 
two heuristics may contradict or give different 
answers to the same question and still be useful; 
heuristics permit the solving of unsolvable prob- 
lems or reduce the search time to a satisfactory 
solution; the heuristic depends on the immediate 
context instead of absolute truth as a standard of 
validity. 

Learning to use heuristic strategies is necessary 
but not sufficient to ensure com]>eient problem- 
solving performance. Schc.Mifeld asserts that these 
equivocal results have occui led because the com- 
plexity of heuristic strategies, and the amount of 
knowledge needed to implement them, have been 
undeiestimated in three ways |l 2|: 

(1) Typical descriptions of heuristic strategies 
(examining special cases, for example) are 
really labels for categories of closely related 
strategies. 

(2) The implementation of heuristic strategies is 
far more complex than at first appears. 

(3) Although heuristic strategies can serve as 
guides to relatively unfamiliar domains, they 
do not replace subject matter knowledge or 
compensate easily for its absence. 

1-lcuristics for learning how to learn are likely to 
be a promising area of research and development. 
If the teaching of heuristics is to be effective, 
however, it must focus not only on the heuristics 
themselves but on their a])plication in a variety of 
contexts, that is, on when and where to apply them. 



CONTRIBUTIONS OF KNOWLEDGE 
ENGINEERING 

Some of the contributions of knowledge 
engineering (and of the broader field of cognitive 
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science) to metalearning include models of the 
learner, experi-novice differences, acquisition of 
expcriise. and knowledge structure and represen- 
tation. 

Models of the learner 

Bruner |13| outlined five models of the learner 
that serve as a useful guide to gaining a perspective 
of ihe conlribuiion of cognitive science to meta- 
learning. The five models are Tabula rasa, Hypo- 
thesis generator, Nativism, Constructivism, and 
Novice-to-expert. The central notion of Tabula 
rasa "one learns from experience'' rests on the 
premise that experience writes on the wax tablet of 
the mind. According to this model such order as 
there is in the mind is a reflection of the order that 
exists in the world. Hypothesis generator learner 
models react against the passive, tabula rasa 
models and propo.se that the learner, rather than 
being a creature of experience, selects what is to 
enter the mind. Nativism theories share one central 
concept: mind is inherently or innately shaped by a 
set of underlying categories, hypotheses, and forms 
of organizing experiences. The opportunities to use 
and exercise the innate powers of mind is all. The 
tenet of ConsirucfivLsm is that the world is not 
found, but made, and made according to a set of 
structural rules that are imposed on the flow of 
experience. The recently developed Novice-to- 
expert view operates within domains and begins 
with the premise that, if you want to develop 
learning St rategies. find an expert and examine him, 
then figure out how a novice can become one. 
Computer simulations are often used to attempt to 
identify transforms and heuristics that will allow a 
novice to become an expert. 

Bruner |14| offered his synthesis of the general 
models that we store in our heads that guide our 
perception, thought and talk by saying that **lhcy 
appear to be diverse, rich, local, extraordinarily 
generative". Bruner also discussed two modes of 
thought, two modes of cognitive functioning, each 
providing distinctive ways of ordering experience, 
and of cimsiruciing reality. One mode, the paradig- 
matic or logico-scientific one, is familiar to 
engineers and scientists. The other, the narrative 
mode, deals in human or human-like intention and 
action and the vicissitudes and consequences that 
mark their course. The importance of the narrative 
mode lies in Bruncr's claim that "great fiction, like 
great mathematics, requires the transformation of 
intuiiions into expressions in a symbolic system— 
natuial languaec or some arlificialized form of it 
(p. 15)". 

NovicC'to-experi 

Enormous differences of degree and of type have 
been observed in the aj^proach taken by experts, 
novices and umnstructed students in .solving prob- 
lems. Novices ask questions such as "What formula 
do 1 know that relates what's given with what Tve 
been asked to find?" They (luickly move to a 
calculation phase and seldom reflect (at least 
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overtly) on what iheyVc doing. Experts "sk C|»e>>- 
ons such as "What are the general pnnc.p'" ^ 
aonW" They spend more time thmkmg about the 
pffiem! ask'in'g themselves c.uesuons and co.^ 
mentinc on their understandmg of the pr^'^'^rn- 
Schden'feld 115| mapped tl- .-themauca p oh- 
lem solving activities of novices and experts on 
numerous problems and noted major diffeien e 
described above. Similar results to those noted 
bv Schoenfeld were found for uninstrxicted stu- 
dents novices and experts solving physics 
problems by Cliampagne ci nl. \ 1 fij. 

Modeling the problem-solvmg methods o 
experts and utilizing these models as instructional 
aids s a very attractive approach, but it is compli- 
cated by what experts arc able to tell about what 
thev know. Furthermore, although nov,ce-to- 
expert models are the principal ones followed by 
cognitive-science researchers it is important o 
keep in mind lhal there are many other models of 
the learner. 

Acquisition of expertise . ■ ■ , ., 

Expertise appears to be acquired throug 
process consisting of three more o. less disl net 
stages of learning 1171. In the first phase, often 
termed the stage of cognition or thought studen s 
learn from instruction or observation what know- 
ledge and acticns are appropriate. In the second 
phase, often termed the "''soc.ative phase of 
learning, students practice (with feedback) the 
relationships discovered or taught in phase one 
until they become smooth (nueni and efficient) and 
accurate (proficient). In the third phase, termed the 
stace of automaticity. relationships are "compiled 
thrtuph overpractice to the point where.they can 
be dtne without large amounts of cognitive 

'Te^'s and Dreyfus 118 19] extended and 
elaborated on these distinct phases of skill acqiu.^i- 
tion and proposed five stages-novice, advanced 
beginner, competent performer prolicient pc 
former, and expert. The novice knows has. fact 
about a subject and context-independent rules tor 
usin" those facts. Tlie advanced beginner can use 
examples to formulate rules for action and can take 
context into account. The competent perforrner is 
nersonally involved, goal-oriented, and is able to 
reason analytically -and act without conscious 
thought about the rules. The proficient performe 
can recall whole situations and apply them without 
having to decompose them into smaller com- 
poneius. The expert makes little conscious use o 
analytical reasoning, has lillle awareness of tlu 
skill is fully involved in the situation, and seems to 
operate by visualizing and manipulating whole 
objects and .situations. 

Performance on tasks at the expert evcl is 
usually smooth and proficient; however, the pro- 
cesses used by experts in their performance art 
oenerally not available to conscious i'^^ueness 
Polanyi 1201 refers t(. this type of knowledge as 
'tacit" ohnson 1171 has termed this lad the 



"paradox of expcrtisc"-thc very knowlcugej^e 
wish t(^ represent in a comi^uter progrsim. as wui as 
the knowledge we wi.sh to teach others, often turns 
(Hit to be the knowledge that individuals are least 
able to talk about. 

Knowledge sinuiuir and represeiiKiiion 

Research in cognitive science has contnbutecl to 
our understanding of knowledge structure, repre- 
sentation and construction. Researchers have 
experimented with production systems They have 
developed inference procedures that involve 
forward-chaining and backward-chaining to act on 
knowledge bases. The modeling of le^.'-"'"^- 
ever turned out to be much more difficult than 
expected. For example, sequential readiness 
assumptions may hold for some simple tasks and 
for young children: however, adolescent and adult 
structures of knowledge and individual differences 
are often uneven and nonlinear 1211. 



CONSTRUCTING KNOWLEDGEBASES 

Outlines and texts are the main methods that 
students use for organizing knowledge externally. 
These approaches appear to ser/e their purposes 
since the majority of student learning invo ves rote 
memorization. Meaningful learning, on the other 
hand, requires more powerful representation 
strategies. A concept map (type of spatial learning 
strategy) summarizing various forms of knowledge 
representation is shown in Fig. 1 . 

Concept maps require representation of rela- 
tionships between concepts: they fac. uate ab- 
straction and deep processing. Unlike more 
a.nient-depcndent techniques (matrixing How- 
charting, constructing pictures or graphs, lor 
example) these systems can be used in a wide vai- 
ietv of texts. Concept maps, according to Novak 
and Gowin 151. are intended to represent meaning- 
ful relationships between conceits m the torm oi 
propositions. In its simplest form, a concept map 
would be just two concepts connected by a linking 
word to form a proposition. 

Outlines and concept maps differ m three main 
wavs ISl- (1) concept maps show key concepts and 
propdsitions in very explicit and concise language 
whereas outlines usually intermix instructional 
examples, concepts and propositions in a matrix 
that may be hierarchical, but fails to show the 
suiK-rordinate-subordinate relationship between 
kcv concepts and propositions; (2) concept niaps 
:,re concise and show the key ideational relatuin- 
ship in a simple visual representauon; and (3 
concept mails visually emphasize both hierarchical 
relationships between concepts and propositions 
and cross-links between sets of concepts and pro- 
positions. , , 1 / . 

The process of constructing knowledge bases (oi 
representations) is very powerful lor assisting 
students in learning h(nv to learn as will be 
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overtly) on what iheyVe doing. Experts nsk ques- 
tions such as "What are the general principles that 
apply?" They spend more time thinking about the 
problem, asking themselves questions, and com- 
menting on their understanding of the problem. 
Schoenfeld \i5\ mapped the mathematical prob- 
lem solving activities of novices and experts on 
numerous problems and noted major differences 
as described above. Similar results to those noted 
by Schoenfeld were found for uninstrucied stu- 
dents, novice.^ and experts solving physics 
problems by Champagne ci ai \ 1 6|. 

Modeling the problem-solving methods of 
experts and utilizing these models as instructional 
aids is a very attractive approach, but it is compli- 
cated by what experts arc able to tell about what 
they know. Furthermore, although novice-to- 
expert models are the principal ones followed by 
cognitive-science researchers it is important to 
keep in mind that there are many other models of 
the learner. 



•^paradox of expertise"— the very knowledge we 
wish to represent in a computer program, as well as 
the knowledge we wish to teach others, often turns 
out to be the knowledge that individuals arc least 
able to talk about. 

Kftowlcdi^c sit uaurc and rcprcsettutiinn 

Research in cognitive science has contributed to 
our understanding of knowledge structure, repre- 
sentation and construction. Researchers have 
experimented with production systems. They have 
developed inference procedures that involve 
forward-chaining and backward-chaining to act on 
knowledge bases. The modeling of learning, how- 
ever, turned out to be much more difficult than 
expected. For example, sequential readiness 
assumptions may hold for some simple tasks and 
for young children: however, adolescent and adult 
structures of knowledge and individual differences 
are often uneven and nonlinear |21 1. 
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Acquisition of expcnisc 

Expertise appears to be acquired through a 
process consisting of three more or less distinct 
stages of learning |17|. In the first phase, often 
termed the stage of cognition or thought, students 
learn from instruction or observation what know- 
ledge and actions are appropriate. In the second 
phase, often termed the associative phase of 
learning, students practice (with feedback) the 
relationships discovered or taught in phase one 
until they become smooth (fluent and efficient) and 
accurate (proficient). In the third phase, termed the 
stage of automaticity, relationships are "compiled*' 
through overpractice to the point where they can 
be done without large amounts of cognitive 
resources. 

Dreyfus and Dreyfus |I8, 19) extended and 
elaborated on these dislinci phases of skill acquisi- 
tion and proposed five stages— novice, advanced 
beginner, competent performer, proficient per- 
former, and expert. The novice knows basic facts 
about a subject and context-independent rules for 
using those facts. The advanced beginner can use 
examples to formulate rules for action and can lake 
context into account. The competent performer is 
personally involved, goal-oriented, and is able to 
reason analytically and act without conscious 
thought about the rules. The proficient performer 
can recall whole situations and apply them without 
having to decompose them into smaller com- 
ponents. The expert makes little conscious use of 
analytical reasoning, has little awareness of the 
skill, is fully involved in the situation, and seems to 
operate by visualizing and manipulating whole 
objects and situations. 

Performance on tasks at the expert level is 
usually smooth and proficient; however, the pro- 
cesses used by experts in Iheir performance are 
generally not available to conscious awareness. 
Polanyi |20| refers to this type of knowledge as 
"tacit" Johnson |171 has termed this fact the 
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CONSTRUCTING KNOWLEDGE BASES 

Outlines and lexis are the main methods that 
students use for organizing knowledge externally. 
These approaches appear to serve their purposes 
since the majority of student learning involves rote 
memorization. Meaningful learning, on the other 
hand, requires more powerful representation 
strategies. A concept map (type of spatial learning 
strategy) summarizing various forms of knowledge 
representation is shown in Fig. I . 

Concept maps require representation of rela- 
tionships between concepts: they facilitate ab- 
straction and deep processing. Unlike more 
content-dependent techniques (malrixing, flow- 
charting, constructing pictures or graphs, for 
example) these systems can be used in a wide var- 
iety of texts. Concept maps, according lo Novak 
and Gowin 15], arc iniended lo represent meaning- 
ful relationships beiween concepts in the form of 
propositions. In its simplest form, a concept map 
would be just two concepts connected by a linking 
word to form a proposition. 

Outlines and concept maps differ in three main 
ways |5|: (1) concept maps show key concepts and 
propositions in very explicit and concise language 
whereas outlines usually intermix instructional 
examples, concepts and propositions in a matrix 
that may be hierarchical, but fails to show the 
superordinate-subordinate rehitionship between 
key concepts and propositions; (2) concept maps 
are concise and show the key ideational relation- 
ship in a simple visual representation; and (3) 
concept maps visually emphasize both hierarchical 
relationships between concepts and propositions 
and cross-links between sets of concepts and pro- 
positions. 

The process of const ructing knowledge bases (or 
representations) is very powerful for assisting 
students in learning how to learn as will be 



379 



36S 



K, A. Smith 



Knowledge 
Representotions 



Active Leorning 
ond processing 



I 
I 
I 
I 

I 
I 




I 
I 
I 



Control structure 
mctocognition 





Leorning 






thot t% 


Acquisition.storoge 
ond/of use of informotion 




I Written mode | |Orol mode 



l-ig. l.C'DiJCcpl injip l()r**ct>i)suucuini km>wlcilgcb;isc>' 



described Inter. Addiiioniil siraicgies for repre- 
.scnlini' knowlcduc arc described in Sniilii ci aL 
\22\. ' 

Expert systems 

Our work in the area of rc(|uiring siudenls lo 
construct explicit knowledge representations 
hcpxn when we introduced the idcu of students 
building small expert systems. Our primary pur- 
pose was to familiarize ihem with this aj-jproach in a 
course on the application of operations research 
techniques in engineering. The introduction of this 
idea had several unanticipated side effects. One, 
the students were much more enthusiastic than we 
had expected. Two, they mastered content that we 
had not expected that they would master. Three, 
they formulated rules for design and decision- 
making that showed they had not only reviewed a 
large amount of information, Init that they had 
reviewed it selectively and purposefully. The 
outcome of this procedure is described briefly 
below and is described in more detail in a recent 
article by Starfield ctal. |23|. 

A small expert system shell has been an in- 
dispensable part of the way in which wc have used 
knowledge bases as a teaching tool. The shell, 
described in Starfield ctaL |24| is written in Pascal 
and runs on a personal computer with 12tSK 
memory. The knowledge bases constructed by the 
students are stored in text files, according to a 
.simple and llexible format. These files are input 
data for the shell, which will read, parse, check and 
interpret the text. The shell then permits users to 
interact with the knowledge l">ase. 

Th'? knowledge base itself is divided into three 
parts. First, there is a set of numbered decisions. 
The second part consists of a series of questions 
which aim to solicit the information nccessarv to 
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.select an appropriate deci.sion; associated with 
each question is a limited number of an.swers. The 
third part consists of a set of production rules. Each 
rule has the formation 

IF <condition> THEN <decision>, 

where the condition is a Boolean expression 
relating to an.swers lo the questions and perhaps to 
decisions. 

In a typical project suidents built a knowledge 
base to .select an url-^an transportation .system. The 
decisions in that case consisted of a list of trans- 
portation alternatives, such as 

Deci.sion 1 buses 
Deci.sion 2 light rail. 

The questions r .ted to the range of lane capacity 
needed, the maximum po.ssihie investment, the 
speed retjuired, the levels of acceptable noi.se and 
pollution, etc. A typical question might be 

Question 3 "What is the range of lane capacity you 
need?" 

An.swer 1 "Between I and 2(),()()() spaces 
per hour" 

2 "Between 8000 and 15.000^'. 

An example of a production rule might Ix' 

If (02 Ans.l) and (not Dec4) and (05 Ansl or 
0^1 Ans2) THEN Dec:. 

In the interaction between a user and the expert 
system shell, the .shell will ask questions, record and 
remember the user's lesponses. and systematically 
test out the rules until i; finds a rule that is valid. It 
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will then print out the appropriate decision. An 
additional, and from the instructional point of view, 
important feature is that the user can at any stage 
ask the shell "why?**. The knowledge base may 
contain reasons (text strings) that are associated 
with each question and each nile. These should 
explain, briefly, the reasoning behind the questions 
and rules, respectively. When a user asks "why?" he 
is first given the reason associated with the current 
question. If he asks "why?" again he is then given 
the reason associated with the rule that the shell is 
currently testing. 

We have found the following sequence to be 
effective in the classroom: 

(1) First we introduce the concept of an expert 
system and knowledge base, explain the 
structureof the production rules,and demon- 
strate a small system on the computer. 

(2) We then divide the class into groups of two or 
three and ask (hem to suggest topics that 
would lend themselves to this kind of 
approach. The ensuing discussion highlights 
the differences between topics that are 
suitable and those that are not. 

(3) Each group is then required to construct a 
knowledge base as a homework assignment 
overaperiodof 1 or2weeks.Thestudentsare 
told to pay particular care to the explanation 
facility and are required to implement and 
demonstrate their work, using the shell. This 
allows faculty and fellow students achance lo 
critique the assignments. 

Ourexperience is that students adapt very quickly to 
thisformal structure, learn to exploit it, andget a very 
real sense of achievement when they implement 
their work. The structure forces them to approach 
their problem in a pragmatic and purposeful 
manner. It guides them into thought processes that 
they may not have previously encountered in a 
structured environment and leaches them lo think 
explicitlyinways that will be essential to themin their 
professional careers. 

Dar f bases ami spreadsheets 

The expert system described above is an example 
of *'build/run" software. It enables students to 
construct a knowledge base, operate on it, and 
examine the results. Standard database and spread- 
sheet programs facilitate the same operations. Stu- 
dents can compare, merge and test information in a 
database; in adtiition, a spreadsheet allows them to 
introduce and simulate mathematical relationships. 
A successful spreadsheet application must be care- 
fully constructed of knowledge and rules (i.e. values 
and formulae) in order that one might laterobserve a 
ripplirgeffeciofcolumn by column output of the cal- 
culated values. Similarly, a database must be con- 
structed before various sorting and merging 
functions can be in voked. These programs provide a 
means lo represent and manipulate knowledge. 

Spreadsheet programs are used increasingly 
outside the traditional business domain. For 



cxample,sprciidsheetshavebccnuscdbystudcntsin 
mine network analysis to simulate air flow, wherein 
known values of air pressure at selected network 
junctions (entered as real values) and physical laws 
(entered as formulae) express the interdependence 
of i\eiwork junctions and network branches, 

INSTRUCTIONAL USE OF KNOWLEDGE 
REPRESENTATIONS 

The strategies described above are lechniciues for 
externalizing concepts and propositions. However, 
learning the meaning of a piece of knowledge 
recjuircs dialog, exchange, sharing, and sometimes 
compromise. Meanings can be shared, discussed, 
negotiated and agreed upon. When spatial learning 
.strategies are used in groups of two or three students, 
it can serve a useful social function and also lead lo 
lively classroom discussion. Cooperative learning 
groups, an active learning technique described by 
Smith |25I andSmilh ctaL [26, 21\ has shown similar 
positive outcomes. Preparing lo leach or luior 
another, whether or noi any leaching is actually 
done, results ingreaier achievement and likingoflhe 
subjectand other class members |28, 29|. Cognitive- 
psychology researchers have shown thai the two 
principal contributors to the development of 
expertise are talking with peers and preparing to 
leach 13()|- 

The learner must engage in active analysis of the 
structure in order to construct a spatial rcpresenla- 
tion. Spatial strategies may be effective not because 
they provide an image, but rather because, by 
constructing a graphic representation, the learner 
carries out activities such as analysis, encoding and 
organization that are themselves effective regardless 
of whelher or not they result in a spatial representa- 
tion. 

In addition to training and encouraging students 
10 construct knowledge bases lo assist in learning, 
instructors could incorporate a variety of forms of 
knowledge bases in their lectures or handout 
materials. Day 13l| described modifying lecture 
materials to incorporate some of these ideas. The 
con.struction of knowledge bases can be incor- 
porated in evaluation procedures with, for example, 
a scoring procedure developed for concept map- 
ping. We have required students to build the 
knowledge representation for expert systems in 
exnms. 

Many recognize microcomputers as a revolution 
in education, not to supplant traditional educational 
processes, but rather to supplement ihcm by 
allowingstudents to experiment with manydiflcrcnt 
situations and to "instruct" the machine rather than 
be "instructed" |32. 33|. Programs whicherutbleonc 
to modify data and quickly recalculate are excellent 
tools for sensitivity analy.sis and encourage a deeper 
understanding of the behavior of the system being 
studied. OiK*e a knowledge representation is con- 
structed, one is free lo operate on it with the tools ol 
the particular package using a "WHAT IF'/" 
approach. 
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SELECTION OF KNOWLEDCIE 
REPRESENTATION STRATEGIES 

Mclalcarning requires a eaj-^abilily for examining 
one's own knowledge and IluuiglUs and llien 
modifying Ihcm aeeordingly. Tlie "eonlrol slrue- 
Uirc'Mhalcanneeomplisii iliis modinailion isealled 
••melacognilion" in liie eognilivc-seience literature. 
•Flavell's |34| generally aecepled dennilion of 
mclacognilion is as follows: 

"Metacognilion refers lo one's knowledge con- 
cerning one's own cognitive processes and pro- 
ducts or anything related to them, e.g., the 
learning-relevant jiroperties of information or 
data. . . . Metacognilion refers, among other 
things, to the active monitoring and consequent 
regulalionand orchestration of these processes in 
relal ion to the cognitive objectives on which they 
bear, usually in the service ofsome concrete goal 
or oi)jcetive'* (p. 232). 

Melacognition has two separate but relatctl aspects: 
( I ) knowledge and beliefs about cognitive phenom- 
ena, and (2) the regulation anti control of cognitive 
actions |34|. Since there are numerous strategies 
availableloeachlearner,theapproachlhatisneeded 
is lo help the student choose appropriate strategies 
for each learning task. Bruner argues against 
promoting only one model of the learner and 
suggests instead that the best approach is a reflective 
one that allows one to "go meta'\ He concludes: 

"Any learner has a host of learning strategies at 
command. The salvation is in learning how to go 
about learning before getting irreversibly beyond 
the point of no return. We would do well to equip 
learners with a menu of their possibilities and, in 
the course of their education, to arm them with 
procedures and sensibilities that would make it 
possible for them to use the menu wisely." 

McKeachie |36| points out in a critique of spatial 
learning strategies that he has difficultyjustifyingthe 
amount of time neeiled to teach students how to 
iniplement these strategies. In contrast, our ex- 
perience is that it takes very little time for students lo 
learn how to operate within the formalism of a rule- 
based knowledge system, or to exploit the facilities 
provided by adatabaseorspreadsheet package. One 
might speculate that, if spatial learning strategies 
were automated, they would become a far more 
efrcciive teaching tool. They would then share the 
advantages of the computer-aided representations 
discussed here, namely, that they provide a frame- 
work for the student, an intellectual "jungle jim" 
which encourages them: ( I ) to review (information 
or data) and select from it; (2) lo think about what is 
imporiani and what is secondary; (3) to hypothesize 
inierconneclions, eonsec|uences and relationships; 
(4) to test those hypotheses and explore sensitivity; 
and (3) to communicate what they learn. 



IMPLICATIONS FOR ENGINEERING 
EDUCATION 

In real-world engineering practice, problems do 
not present themselves lo the practitioner as givens. 
Problem formulation, the process by which we 
denne the decisions to be made, the ends to be 
achieved, and the alternative means which may be 
chosen, is neglected in much of engineering educa- 
tion. Problem formulation requires the capability to 
learn how to learn and to reflect-in-action. Accord- 
ing to Schon |4| refleclion-in-action exemplifies 
professional activity: 

"When someone reflects-in-action, he becomes a 
researcher in the practice context. He is not 
dependent on the categories of established theory 
and technique, but ccmstructs a new theory of the 
unique case. His inquiry is not limited lo a 
deliberation about means which depends on a 
prior agreement about ends. He doas not keep 
ineans and ends separate, but defines them 
interactively as he frames a problematic situation. 
He does not separate thinking from doing, 
ratiocinating his way to a decision which he must 
later convert toaction.Becausehisexperimenting 
is a kind of action^ implementation is built into his 
inquiry. Thus refiection-in-action can proceed, 
even in situations of uncertainty or uniqueness, 
because it is not bound by the dichotomies of 
Technical Rationality " 

In Educating flic Reflective Proctitioner Schon |37| 
describes his approach to the development of 
professional practice skills. He writes: 

"Designing, l')oth in its narrower architectural 
sense and in the broader sense in which all 
profession practice is designlike, must be learned 
by doing. However much students may le-drn 
about designing from lectures or readings, there is 
a substantial component of design competence— 
indeed, the heart of it— that they cannot learn in 
this way. A designlike practice is learnable but is 
not teachable by classroom methods. And when 
student.s- arc helped to learn de»sign, the inter- 
ventions most useful to them are more like 
coaching than teaching— as in a reficctive practi- 
cum" (p. 1 57). 

Champagne's |38| research on scientific know- 
ledge and the mechanisms by which it is learned 
indicates that scientific knowledge is complex and 
often tacit, learners construct understanding; refine- 
ment of personal theories about the natural world is 
an important part of the learning of science, and 
social interaction is a powerful mechanism for 
producing cognitive change. 

The cognitive-psychology research confirming 
the importance of peer interaction in the le<\rning 
process has important implications for engineering 
education. Meaningful learning and especially 
learning how to learn is enhanced by talking with 
peers and preparing to teach others. They are 
essential aspects of learning. Social psychologists 
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(Lewin, Deulsch and the Johnsons) have" been" 
advocating this practice for over 40 years. 

A key, therefore, to learning how to learn is to get 
students involved in the construction of knowledge 
representations |22, 39|. Getting students involved 
in meaningful problems is a powerful means for 
developing talent |34, 40], We must, in addition, let 
students struggle with problems, especially at the 
formulation stage* We can provide them with some 
learningstrategies to make their task a bit easier (42- 
44|. For example, Schoenfeld acts as a roving 
consultant while the classbreaks iniosmall groups lo 
work on mathematics problems. He has found that 
asking the following three questions promotes the 
development of meiacognilive skills |45|: 

( 1 ) What (exactly) are you doing? 
(Can you describe it precisely?) 

(2) Why are you doing it? 

(How does i! fit into the solution?) 

(3) How does it help you? 

(What will you do with the ou icome when you 
obtain it?) 

Heller and Hungate [46] conclude that students 
need lo become better able lo reason qualitatively 
about problems and ?o know when and how to 
perform the many component procedures. They 
suggest several ways students' attention could be 
turned to learning these particular activities, includ- 
ing: (1 ) make tacit processes explicit, (2) gel studen is 
talkingaboutprocesses(an idea proposed by Bloom 
and Broder in 1950 |47), (3) provide guided 
practice, (4) ensure that component procedures are 
learned well, (5) emphasize both qualitative under- 



standing and specific procedures, and (6) tesl for 
understanding and reasoning processes. 



CONCLUSION 

The development of higher cognitive skills that 
enable students lo be independent learners and 
independent, creative problem-solving users of 
their knowledge is a very important goal for 
educators. Providing students with an active le;irn- 
ing environment where they can gel involved with 
the material lo be learned in a mutually supportive 
situation with other people and providing them with 
tools such as Iheones described here will contribute 
to meaningful learning. 

Even if learning, thinking and problem-solving 
slralegies, whether general or specific, arc shown to 
exist, il might nol be possible (o leacli ihem directly. 
Perhaps they must spontaneously emerge as con- 
sequence of substantial experience. The current 
conception is that metacognition— conscious 
awareness of and control of cognitive processes- 
emerges only as knowledge and skills in a particular 
domain become quite well developed. At the very 
least, it should be possible lo select and design 
experience to result in a more rapid and complete 
emergence of such skills. A key to the success of 
developing students' skill at using these strategies is 
for faculty to incorporate them in their handouts, 
exercises,lectures,assignmentsand exams. We must 
also be willing lo accept that il may be the idiosyn- 
crat'c, coven concept meanings that are the 
principal factors in most human learning [48, 49], 
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Testing Intellectual Skills in Engineering; 
Test Design Methods for Learning* 
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This study examines classroom testing from the perspecti\^ of psychological learning. Four 
Icctual skill levels are defined, based upon basic and classic research on human lea 
pxvchology. These intellectual skill levels can then serve as the foundation for design ' 
questions. Additional psychological learning variables are considered, and how testing ii 
with these variables. Finally, the results of a survey on testing methods used by engineering ft 
are presented, • « ; 
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TESTING in engineering courses is typically 
viewed by university faculty members as simply a 
method of measuring course content students 
acquire, and providing grades on academic 
achievement in an engineering major. However, 
testing can be much more than this. Since 1981, the 
College of Engineering at the Pernisylvania Slate 
University has conducted, teaching seminars for 
engineering faculty, and ,a course on teaching 
effectiveness for engineering teaching assistants. 
One of the major topics during these seminars and 
course is test development. At the start of the test 
development seminar faculty members .and teach- 
ing assistants are asked to write a sentence or two 
describing the major importance of classroom test- 
ing in engineering. Of the 805 faculty members and 
teaching assistants asked to respond over the years, 
87% indicated that tests are for giving grades and 
evaluating the amount of information acquired. 
Other responses included: evaluating how effec- 
tively minerial was taught, motivating students, and 
providing feedback to students on skills learned. 
While sincere regarding their desire to be effective 
instructors, few engineering faculty members and 
leaching assistants understand the relationship 
between testing and psychological learning, and the 
overall value of testing for achieving specific intel- 
lectual outcomes. } 

This paper will furtiier examine the fundamental 
relationships of testing and psychological learning. 
Tlie issue of a test as an independent variable in the 
learning process was considered in our first paper 
on testing ll). Tliis issue will be elaborated upon 
here with a more detailed discussion of psycho- 
logical learning, or intellectual skill levels, and 
classic research findings related to these topics 
from the research on concept learning and problem 

' Paper accepicd 23 Au^iust 1 988. 
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solving. A' number of sample test problems will 
then be presented and analyzed in terms of intel- 
lectual skill levels. A number of references will jthen 
be provided on developing effective tests, fron] key 
researchers in the area pf testing theory and! test 
construction. Finally, survey results will be jpre- 
sented and discussed, from a survey evaluating the 
typical testing methods I used by the engineeriiig 
faculty. This survey examined testing methods, 
intellectual skills, and types of engineering couVses. 



RELATING TESTS TO PSYCHOLOGICAL 
LEARMNG VARIABLES 



A basic psychological variable (learning vari- 
able) that the test affects is the variable or motiva- 
tion. Of course, motivation is a fundamental 
psychological variable in the learning process. 
Without motivation, little or no learning takes 
place. The variable of motivation cari be consi- 
dered a continuum, ranging from sleep to a high 
level of state anxiety. However, the type of motiva- 
tion that is most related to classroom testing is 
achievement motivation [2, 3). Achievement moti- 
vation research attempts to explain the basics of 
goal seeking behavior, and what causes various 
forms of this behavior. A detailed discussion of 
achievement motivation can be found in Atkinson 
and Feather [4). Achievement motivation research 
results are complex, indicating that a great depl of 
environmental and subtle psychological fadlors 
stimulate the need to achieve certain goals, sjieci- 
fically academic goals, as Travers explains, p. ^ 34: 

The students in a class may appear bored and 
listless, but this does not mean that they do not 
have potential for showing achievement moliva- 
tion. Perhaps their specific needs are not being 
aroused [5). , 

The type of achievement motivation the class- 
room test seems to stimulate most often is the 
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desire to acquire favorable grades. Unfortunately^ 
grades are not the most appropriate type of 
achievement motivation, but students are highly 
motivated by grades on tests and major assign- 
ments. This level of motivation is known as external 
motivation, discussed in the locus of control 
icxcarch |6|.Thc overall outcome of a professional 
degree should be to lead to internal motivation |7|, 
or the desire to be a productive and competent 
engineer for that reason alone. Hut, in reality, most 
undergraduate *;tudent5 are stimulated to learn 
because of grades. Therefore, the test directly 
affccls nu)!ivaiion lo learn, by stimulating achicve- 
mciii motivation, to acquire favorable grades. This 
relationship should be positive, and not a negative 
or punishing type of motivation. Students will tend 
lo study more completely, and more seriously, 
close to the lime of major tests |8|. The frequency of 
testing will then directly manipulate the amount of 
.serious study students will engage in. So, the 
p.sychological variable of achievement motivation 
can be directly manipulated by frequency of test- 
ing. It is likely that giving only a mid-term and final 
examination is not appropriate since this contri- 
butes to a cramming study strategy by students. 
More frequent testing would lead to a distributed 
study strategy by students, and keep their motiva- 
tion levels for learning new material at a constant, 
probably optimal level, rather than peaks of high 
and low motivation. 

The second psychological variable tests can 
control is the level of intellectual skill at which 
students will acquire course content. It is not 
surprising -to engineering faculty members to 
ob.serve a wide variety of understanding levels in a 
class, on any given topic being presented. For 
example, some students seem to simply memorize 
' ~w information, while others appear to have a 
much ^deeper' understanding of the material. This 
depth of learning phenomenon is in reality a mani- 
festation of the intellectual skill level students are 
psychologically operating at with the new material. 
The lest can be used to directly control this intel- 
lectual skill level students attempt to acquire. For 
example, if a test usually asks students to memorize 
basic facts, such as formulas, symbols, and defini- 
tions, chances are the majority of students will 
learn at the factual intellectual skill level with 
course material. If tests provide the facts, but then 
require students to apply ntw rules they should 
untlersiand, chiince are, students will master 
material at the rule application level. Of course, 
instruction must be at the appropriate intellectual 
skill level to effectively prepare the student for the 
testing situation |9). 

There are a number of researchers who have 
attempted lo define intellectual skill levels |1 0, 11, 
] 2. 1 3). For many teaching and learning situations, 
their definitions are useful. However, when 
explaining engineering course material a more 
classical approach to defining intellectual skill 
levels, based on research from learning psycho- 
logy, .seems to be applicable |5, 1 4, 1 5). 



Most engineering course content' seems to fit 
four basic intellectual skill levels identified as the 
psychological re.search related to learning. From a 
basic level to a more sophisticated level, these intel- 
lectual skill types are: factual learning, concept 
learning, rule learning, and problem' solving. The 
key is to be able \o define course material at these 
different intellectual skill levels, and then prepare 
test questions at the same level. Subsequently, this 
should stimulate students to learn material at the 
appropriate intellectual skill level] since tests 
require them to perform at that level. I . 

The basic intellectual skill level is factual learn- 
ing. All educated individuals must acquire a vast 
number of facts to be able to perform more 
complex intellectual tasks. Classic ; research on 
problem solving, indicated that more complex 
problem solving tasks required the acquisition of 
factual information, 1 16, 17, 18|.Thelearly Gestalt 
Psychologists spent a good deal of effort explaining 
the differences between faculty memqry and prob- 
lem solving, which they sometimes called drill and 
practice versus meaningful apprehension |19], or 
reproductive versus productive thinJcing j20]. So, 
while facts are the most basic level of intellectual 
skill that can be acquired, they forni the crux of 
much later and more complex intellectual skills. 
Students should be required to learn these basic 
facts in a given area, and test questions of a factual 
type can easily be developed to test this skill. The 
factual level test question can be designed to stress 
recall of information or recognition ofinformation. 
Simple completion items are quite good for factual 
memory testing, as well as matching type test 
questions |1J. Of course, multiple choice items can 
be designed to test factual memory of a recognition 
type. It is more difficult, however to recall factual 
information than to recognize factual information 
stored in memory j21). Therefore, if the facts are to 
be well learned for future use, a recall test quesUon 
may be better, such as a completion type of item. 

The second level of intellectual skill is concept 
learning. There is a good deal of history behind the 
research on concept learning beginning in 1920 by 
Hull. Similar to Hull (221, ^he work b>; Heidbreder 
|23J on concept learning and Bruner, Goodnow 
and Austin |24|, indicated that concepts are classi- 
fication systems humans use to categorize informa- 
tion in the environment, The concept is a definition, 
made up of a listing of attributes. iThis list of 
attributes forms the concept category and once 
these attributes are learned, classifying information 
into and out of the conceptual category is a simple 
task. Conceptual thinking is an innately human way 
of thinking, allowing us to deal with large quantities 
of information encountered in the envirormient 
without putting a burden on memory. In fact, 
without the ability to think in a conceptual way, 
human memory would probably have difficulty 
deahng with even the most simple recognition 
tasks. For example, the conceptual task of identify- 
ing and grouping dogs from other animals, and by 
dog type, is quite simple for normal adults, once the 
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conceptual rule about dogs is learned. ^^"^Pjy 
stated the dog concept is: 'Has four legs, has a head, 
has a body, has a tail-usually, and barks/ Knowing 
this information at a conceptual intellectual level 
eliminates the problem of memorizing all images 
and features of all dogs encountered, to be able to 
recognize a dog. or dog type. Computers on the 
other hand, arc very poor conceptual thmkers. Dui 
have better memory capabilities than humans, so 
thev can carry out pattern maichmg memory tasks 
better than we can. Young children often attempt 
memory methods to try and learn concepts and fail 
to think conceptually, and therefore, have difficulty 
with concept tasks 125]. It is common to observe 
smdents who attempt to use memory instead of 
trying to learn conceptually 126]. We often say that 
these students don't seem to comprehend, or can t 
apply what we taught. The phenomenon being 
observed is simply the different intellectual skill 
levels of. factual learning versus conceptual 

learning. ' | .,11 
Concept tasks are common in most college level 
courses, and should be represented on tests. For 
example, a conceptual learning task in electrical 
engineering is the ability to classify basic eleclrical 
componentslfor building circuits, such as resistors, 
capacitors, (iiodes, transistors, etc. There are a 
wide variety of resistors, of varying capabilities, 
shapes, sizes, and colors, but once the conceptual 
rule defining resistor is learned, classifying what is 
and what is not a resistor becomes a simple 
conceptual task. Similarly, many other types of 
concepts ^ can be identified in most engineering 
courses at the college level. The first step in test 
design ihen.iis to differentiate course content along 
the lines' of what is a fact, concept, rule, and 
problem.'anci develop ter.t items that will stimulate 
the appropriate type of thinking in smdents. How- 
ever, it may'be obvious at this stage, that differen- 
tiating course content and writing test items at cer- 
tain intellectual skill levels and evaluating :.mdent 
thinking, is a hypothetical task at best. This is true, 
but after some practice and evaluating tests devel- 
oped in thislway, it becomes a viable method of test 
development. Since the test developer has only five 
test items to select from: compleiion/short answer, 
essay/long answer, multiple choice, true-false, and 
matching, niost intellectual skill levels can be tested 
with these item types. Except for true-false items, 
all four intellectual skill levels can be addressed 
contingent upon how the test item is designed. For 
the concept learning level, test items should require 
students to be involved with concept identification 
[14|, or classifying information into and out of 
conceptual categories. Conceptual thinking can be 
stimulated by requiring the student to: compare 
and contrast concepts, explain or identify attri- 
butes, identify correct from incorrect attributes, 
formulate and explain the conceptual category. 

The next intellectual skill level is rule learning. 
While the definition of concept learning may seem 
awkward, rule learning is a little easier to deal with 
since engineering course content involves ^ wide 
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verbal, and arc usually a part of problem solving. 

The rule can he a logical rule, such as Kirchhotl s 
Voltage Law or the Periodic Law, or a mathe- 
matical rule, like the independent equations to 
solve an electrical circuit problem. The key, how- 
ever, of the rule learning intellectual skill, is that the 
student cannot simply memorize the rule but must 
know when to apply the rule or when not to apply a 
rule; or in some cases, how the rule should be 
changed to fit a certain application. A common 
learning problem among engineering students is 
that of simply memorizing niles. and not fully 
comprehending how to apply rules. There is a great 
deal of difference in intellectual capability between 
memorizing the rule as opposed to being able to 
apply the rule, given the appropriate situation. Rule 
learning is similar to concept learning, and some 
educational researchers have indicated that com- 
plex rules used in problem solving can incorporate 
several lower-level concepts [271. Rule learning test 
questions should attempt to get the learner to appl y 
the rule and avoid simply recognizing or recalling 
the rule, since this would then become a factual 
intellectual skill. The crux of the rule learning test 
question is that it should deal with application. The 
student should be forced to apply the rule or rules 
in a realistic way. by demonstrating how the rule is 
used, or by explaining the application of the rule, 
and in some cases for mathematical rules generat- 
ing the rule. Multiple choice questions can be 
developed to test this intellectual skill, but they will 
require a good bit of time and thought to develop 
•(1). A very good way to test rule application is with 

' a short answer problem, or completion type test 
item. 

Problem solving is the highest level iniellectuai 
skill. Problem solving involves the incorporation oi 
the three more basic levels of intellectual skills: 
facts, concepts, and rule application. Problem 
solving cannot take place without facts being stored 
in memory and conceptual understanding, and the 
ability to apply rules. In actuality, the intellectual 
skill level of problem solving can be broken into 
three distinct sublevels: solving simple problems, 
solving complex problems, and the most difficult 
problem solving type, identifying a problem. The 
identification of problems is seldom done in class- 
room learning. Most of the learning classified as 
classroom learning in undergraduate school 
involves either simple numerical or verbal prob- 
lems, or complex numerical or verbal problems. A 
simple problem can be defined as one involving the 
application of a single concept or rule to find the 
solution. A complex problem can be defined as one 
involving a variety of concepts and rules, and 
perhaps a series of equations, to find a solution. 
The key phrase to both of these definitions of 
problem solving is 'find a solution/ To represent 
true problem solving, the test question should not 
be designed so that the students can respond from 
memory alone. The difficulty with many test 
questions thought to be problem solving is that ilie v 



BEST COPY AVAILABLE 



I 

I 
I 
I 
I 
I 
I 
I 
I 



X Canelos and G. Catchen 



in 



I 

I 
I 



simply require the memorizing of a set of steps, and 
rcpaiting the steps. This algoritliniic style of arriv- 
ing at a sohition is not problem solving, but is 
actually the more basic factual intellectual skill. 
The classic definition of problem solving described 
here can be found throughout the research in 
psychology, related to explaining human problem 
solving behavior |N, 15, 28, 5|. The early Gestalt 
Psychologists conducted much of their research by 
observing and explaining problem solving 
behavior. For example, Katona (29), in 1940, 
explained thai memory tasks fail to allow the 
leiirner lo transfer memorized information to the 
problem solution stage of thinking. Early work by 
Werlheimer [20| yielded results that support the 
notion of differences between memorizing solu- 
tions and problem solving using the application of 
mles. Similar explanations of problem solving can 
he found in very early work by the Gestalts. For 
example, as early as 1925 and 1930, Kohler [30) 
and Maier |32] documented their work on problem 
solving, indicating differences between the intel- 
lectual skills of memorizing solutions, versus 
problem solving by deriving unique solutions. 
Later Gcstalls, using more experimental metliods, 
noted that memory skills alone do not allow for 
complex «nnd creative solutions to problems [32, 
33), and in some cases memory problem solving 
can interfere with seeing a solution. However, tliis 
information contrasting memory skills versus 
problem solving skills, is probably not a great 
surprise to most engineering faculty members since 
they observe this phenomenon in -students at 
chronic levels. To help avoid this difficulty, test 
items should.be designed lo require true problem 
solving involving; recalling some facts, differentiat- 
ing concepts, and applying rules, to arrive at a 
unique solution not encountered before. The test 
items most adept at this type of engineering prob- 
lem solving are long answer problems or essay and 
short answer problems or completion. The rule of 
thumb for the test developer, when writing prob- 
lem solving questions, is lo consider the question: 
'Can my students solve the problem by using a 
memorized solution?' If they can simply write out a 
memorized algorithm, the test question is not a 
problem solving question, regardless of its format, 
essay or completion, or complex multiple choice, 
but is rather a factual question. 

Another powerful psychological learning vari- 
able which is part of testing, but seldom considered 
by engineering faculty members, is that of perfor- 
mance feedback. The effects of feedback on learn- 
ing are well documented in the classjc learning 
theories by Hilgard and Bower )15), and in more 
applied explanations of psychological learning 
related to school learning by Travers )34). Perfor- 
mance feedback can be found in learning theory 
research under the rubric of reinforcements cogni- 
tive feedback, and informative feedback. This feed- 
back variable can have a significant effect upon 
learning, if used appropriately as part of classroom 
testing. In general, the research on feedback 
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concludes that a student will learn more about a 
particular intellectual skill, if more information is 
provided about the quality of the performance, 
including an explanation of what was correct and 
what was incorrect and why. This information 
should then be beyond simple feedback,', like 
posting letter grades, but what is called elaborate 
feedback, Tlie elaborate feedback method can lake 
place by explaining classes of problems students 
tended to answer incorrectly. Such explanations 
should take place in class during post-test reviews. 
If entire groups of lest questions are missed, it may 
be necessary to review that particular material in a 
good bit of detail during a post-test review session. 
Additionally, as part of the post-test review, the 
instructor should take time to explain the relative 
application of important concepts, rules, * and 
problems, to future work in the course. The ela- 
borate feedback method turns the testing function 
of the instructional process into a true learning 
experience, aimed at improving intellectual skills 
for future application; 

SAMPLE TEST ITEMS AND SOME 
ADDITIONAL SUGGESTIONS 

The sample test items will provide some prac- 
tical examples of designing questions at different 
intellectual skill levels. The samples here are from 



courses taught at the Penn State College of 
eering. 



Engin- 



Sample 1 

How many independent equations based on 
Kirchhoffs Current Law may be written for this 
circuit? i 
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How many based on Kirchhoffs Voltage Law? 

I 

The Sample 1, completion test hemj (short 
answer problem) is at the rule application level, 
since it requires the learner lo determine which 
equations apply lo the circuit problem and how 
many. The KCL and KVL equations are being 
defined as rules that can be applied to solve the 
problem, or in this case analyze the problem. It 
would be difficult lo respond simply from rriemory, 
unless this exact circuit was memorized, including 
the accompanying equations, I 

Sample 2 \ 

The layout illustrated in the figure beIo\^| repre- 
sents which of the following circuits? | . 

a. CMOS Inverter ' : 

b. NMOS NOR Gate I 

c. Dual NMOS NAND Gate j 

d. Dual NMOS Inverter i . 
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Sample 2 represenls a multiple-choice test item 
al ihc concept learning level. The circuits are 
defined as concepts, since ihey are made up of a 
definition formed from a set of attributes. Tlie task 
in Sample 2 is to analyze the attributes of the circuit 
in the figure, and determine what category of circuit 
the example falls onto. While memory would help 
with this task, memory alone would not allow for a 
correct conceptual analysis. 

Sample 3 
Tlie term 'Metal Lifl-OfT refers to: 

a. Reliability problem in IC metallization layer 

b. Processing sequence which defines metal 
lines 

c. Selective editing on a full custom layout 

d. Fatigue problem in IC package pins 

Sample 3 is a multiple choice item at the factual 
intellectual skill level. While the fact being tested is 
important if it was committed to memory the item 
can be easily answered. However, while factual 
learning is a basic intellectual skill, if certain facts 
must be learned they should appear on the test. 

Sample 4 
1. 




706.4 keV 



a. Calculate log ft for the decays EC, and 
ECj and give the most likely classification. 



Decay E„ 

(Mev) 



log f log ft classification 



EC, 
ECj 
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b. Assign spins and parities to the unlabelled 
levels in "Fe. | 

c. Now, check the EC classification in part a 
and give the final classification. 

State r 

706.41 keV 
136.5 keV 
14.4 keV 



The test item in Sample 4 is a long answer pr6b-j 
lem (essay) at the problem solving levd. Note here' 
that to evaluate the student's response his or her. 
work would be collected. This is a complex prpb-, 
lem requiring the recall of facts, the ability to apply- 
most mathematical and verbal rules, and compre- 
hend concepts. The mathematical work is obvious 
from the statements in the problem, the rules tojbe 
applied relate to spins and parities, and concepts 
relate to EC-classifications. 

Sample 5 

An NMOS enhancement mode transistor with a 
2 volt threshold and gain factor B « 20 X lO'i is 
operating the saturation region, with Vgs««4V! 
What is the Drain to Source current IDS. • ; 



a. 10 micro-amps 

b. .4 niilli-amps 

c. 1 micro-amp 

d. 40 micro-amps 



Sample 5 is a multiple-choice item at the 
problem solving level. However, this problem is a 
simple numerical problem, and not as complex as 
Sample 4. This problem involves the application of 
the correct equation, the rule in this case, and 
factual information in terms of how to calculate 
data while completing equations ! 

A number of excellent texts are vailable to help 
engineering faculty with the development of effec- 
tive tests. To summarize the work of these authors 
is beyond the scope of this paper but a detailed 
reading of the following would be helpful, parti- 
cularly for new engineering faculty. 

(1) Wilbcrt J. McKeachie, Teaching Tips, A 
Gitidebook for the Beginning College 
Teacher, D. C. Heath (1 986). 
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Chapters 8 and 9 discuss practical sugges- 
tions for lest development and methods of 
assigning grades. 

(2) Elliot A. Weiner and Barbara J. Stewart. 
Assessing Individuals, Psychological and 
Educational Tests and Measurements, 
Little, Brcwn (1984). 

This is a cook book approach to testing, but 
provides numerous example test items. The 
text also contains a discission of grade 
reporting procedures and basic statistical 
methods. 

(3) Lynn Lyons Morris and Carol Taylor Fitz- 
Gibbon, How to Measure Achievement, 
Sage Publications (1978). 

A basic, but practical guide to developing 
classroom examinations, and evaluating test 
results. 

(4) Robert F. Mager, Measuring Instructional 
Results, from the Mager Library Set, Pitman 
Learning (1984). 

For those using an objecti*'e based teaching 
method, this will provide further informa- 
tion on relating lesson objectives to test 
items. Therd are a number of good examples 
of test questions in this text. 



TESTING METHODS USED BY 
ENGINEERING FACULTY 

A survey was conducted at the Penn State Col- 
lege of Engineering during Fall Semester 1987. The 
survey items appear in Table 1. The survey was 
mailed to 226 College of Engineering faculty 
members, and 153 responded. Items 1, 2, and 3 on 
the survey measure were to determine type of 
course taught, and relate this information to type of 
test item typical ly used, and level of intellectual skill 
that test items tended to address. The results from 
items 1, 2, and 3 are graphically presented in 
Figures 1 through 10. Survey items 4 through 9 
evaluated additional issues related to testing in the 
engineering classroom. All survey data are pre- 
sented as percents of those responding to the 
options on each survey item. The data from the 
survey should be representative of other engineer- 
ing colleges, having a large undergraduate enroll- 
ment, and offering comprehensive masters and 
doctoral degree programs. However, the majority 
of undergraduates at Penn State College of 
Engineering are upper level undergraduates, in 
their last two years, or at least starting their fourth 
semester out of eight semesters. Penn State has 21 
branch campus locations at which many freshmen 
and beginning level sophomore students start their 
programs, prior to arriving at the main campus at 
University Park. However, there are beginning 
level students and courses at College of Engineer- 
ing main campus at University Park. One addi- 
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Table !. Sample questions and overall summary data 

i 

1 . Your rcjsponse refers to what type of course? 

(a) Freshman. Sophomore Service Course 

(b) Junior, Senior Service Course j 

(c) Required Course in Major : 

(d) Graduate Level Course ; 

(e) Elective Course in Major I 
' 2. What type of test item do you tend to use most often? 

(a) Essav-lonp answer problem (numerical or 
verbal) | 

(b) Completion-short answer problem (numeri- 
cnl or verbal) 

(c) MuUiple-chnicc , 

(d) True-false ' 

(e) Matching 

(f) Mixed Meihod—Some combinaiion of these 
on a given \tn\ j 

3. Most frequently, the primary purpose of my test 
question is to: i < 

(a) Measure factual knowledge, i.e.. recall of 
information i 

(b) Measure ability to apply mathematical, or 
verb;.! rules, e.g.. use equations ! . 

(c) Simple solutions of numerical problems 

(d) Solutions of complex numerical problems 
A. Do you tend to give partial credit for numerical 

problems? | ; 

947o (a) Yes | ; 

17o (b) No. 

S% (c) Sometimes 

5. Exams are most often graded by: 
71% (a) Professor 

4% (b) TA 

87o (c) TA supervised by the Professor 
I77o (d) Both TA and Professor 

6. Do exams tend to be: 
457o (a) Open book in class 
49% (b) Closed book in class 

6% (c) Take home 

7. How frequently do you give major exams? j 

2% (a) Every 2 weeks 

47o (b) Every 3 weeks 
217o (c) Every 4 weeks 
667n ((I) Evcr>' 5-7 weeks 

7% (e) Once per semester 

8. Do yo'j give any type of quizzes? 
737o (a) No 

277o (b) Yes (24%) (37o) 

(c) If yes: (1) Pre-schcduled (2)' Pop quiz 

9. How d(i you assign grades? ; 
627n (a) Curve, or norm referenced grading 
347o (b) Absolute, or by points equaling grade 

4% (c) Other— standard scores, like T-score 

For results on 1,2. 3. see Figs. I - 1 0. 
Percent re.sponding to each option. 

tional distinction needs to be noted, Ithat is, the 
difference between service courses and required 
courses,i.e„majorareacourses,At Penn State in the 
College of Engineering a service course is a general 
course in engineering taken by all students, such as 
thermodynamics or basic electronics, j 

Figures 1 through 5 provide percentlresponding 
data, and list the test item types. Those test item 
types are: essay or longer answer problem, comple- 
tion or short answer problem, multiple choice, 
true-false, matching, and mixed method. Figures 1 
through 5 compares the results of survey items 1 
with 2, or academic level and typical test item type 
used on a test. Figure 1 indicates that for beginning 
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level service courses faculty members tended lo;use 
completion items about 20% of the time, but used 
long answer problems, 407o of the time u'iih some 
matching and mixed. The mixed method is a aest 
method with a combination of item types Figure 2 
indicated that for service courses at an upper level, 
junior/senior, more essay items are presented on 
tests. Short answer problems or completion items 
are used about the same as the mixed method 
however no matching items are given. A small 
amount of multiple-choice items are use for iihis 
academic level. | 

•Looking at Fig. 3, it indicates that in required 
major area courses, the testing methods tended to 
differ from service courses. There is an equal 
amount of long answer problems (essay) and short 
answer problems (completion) on each test, with 
very little multiple choice or matching. Howevir, a 
mixed method lends to be used more often, 
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probably mixing short answer and long answer 
problems. Interestingly, Fig. 4 indicates that in grad- 
uate level courses most tests are essay, long answer 
problems, with very little completion or mixed 
method. The difference in frequencies between 
graduate and undergraduate lest item types is quite 
significant. In elective courses in a major, the faculty 
tended to use testing methods similar to the junior/ 
senior service courses, with mostly essay and com- 
pletion and some mixed method. 

Figures 6 through 10 compared survey item 1 
with survey item 3, evaluating academic level and 
intellectual skill type the test questions attempted 
to address. Figure 6 considered the service courses 
for freshman and sophomore level. Most test ques- 
tions, 607o, were designed to evaluate rule applica- 
tion, with some lest items involving simple 
numerical problem solving, and complex problem 
solving. For junior/senior service courses (Fig. 7) 
the results are similar to Fig. 6, but Fig. 7 indicates 
that more factual level questions are used. How- 
ever, at the junior and senior level, with service 
courses, the stress is still on rule application and 
problem solving. 

Inierestingly, the required courses in a major, 
Fig. 8, results are similar to results on service 
courses. In required courses, the faculty are stress- 
ing rule application, with a reasonable amount of 
factual knowledge. Figure 9, indicates a similar 
stress on application of rules, but problem solving 
tends to be more complex, which is expected in 
graduate level courses. Finally Fig. 10, indicates a 
stress on the application of rules, but factual 
learning is significantly higher for these elective 
courses. 

Figures 1 through 10 indicate favorable results 
in light of ever increasing enrollments at Penn State 
and probably other similar engineering colleges. As 
enrollments increase, it becomes more difficult to 
evaluate and grade essay and completion type 
items, particularly at the rule application and 
problem solving level. One may hypothesize that 
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increasing enrollments would lead jto factual 
oriented testing with multiple-choice iand other 
objective style test items, but this has not been the 
case. Faculty members are consistently attempting 
10 Slay with rule application and problem solving 
oriented tests, using long answer and completion 
problems, as well as mixed methods. Addressing 
these upper level intellectual skills via testing is 
critical in the engineering curriculum, and should 
not fall victim to enrollment pressures, i 

Survey item 4 indicated that most faculty use a 
partial credit grading method. This is consistent for 
testing at the rule application and problem solving 
level, using mostly essay and completion test item 
types. Item 5, indicates that the professor is not 
delegating lest grading lo the teaching assistant 
most of the time: i 



(1) 



the time the professor does all 



717o of 

grading | 
(2) 257o of the lime the professor and TA do 
grading. i 
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hem 6 indicales lhal open book exams are used 
about half Ihe lime. Most faculty members tend to 
give major tests aboul every 5 to 7 weeks; 87% 
responded in tliese categories. While 5-;7 weeks 
may be a bit too long between tests, item 8 indicates 
that IT/o of faculty give quizzes, thus, adding 
needed study and motivation. Additionally, many 
Penn State College of Engineering faculty members 
give graded homework between major exams, to 
provide performance feedback. Finally, item 9 indi- 
cales that while 62% of faculty graded on a curve. 
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about 34% used an absolqte method based on« 
accumulated points to get a pre-specified grade. 

A future work is planned by the authors on tdst 
ing, to evaluate the testing methods used by olhei! 
engineering faculty at other universities. Plans are:- 
to use the same testing survey measure, but stratifjf 
results by: . 

1 i . ! 

(1) : 2 year technical degree programs, 

(2) \ 4 year bachelor degree programs, 

(3) . Graduate degree programs, 
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